Combining Facial Expressions and Electroencephalography to Enhance Emotion Recognition
Combining Facial Expressions and Electroencephalography to Enhance Emotion Recognition
复制标题
DOI：
10.3390/fi11050105
复制
发表时间：
2019-05-01
期刊：
影响因子：
3.4
通讯作者：
Pan, Jiahui
中科院分区：
文献类型：
--
作者：
Huang, Yongrui;Yang, Jianhao;Pan, Jiahui
情感识别在人机交互中起着至关重要的作用。以往的研究都是将表情信号和脑电信号分别用于情绪识别，但很少有人关注它们之间的融合。在本文中，我们采用了一个多模态的情感识别框架相结合的面部表情和脑电，基于价唤醒情绪模型。对于面部表情检测，我们采用了多任务卷积神经网络（CNN）架构的迁移学习方法来检测效价和唤醒状态。对于脑电检测，两个学习目标（效价和觉醒）分别检测不同的支持向量机（SVM）分类器。最后，采用基于枚举权重规则和自适应Boosting技术的决策层融合方法，将联合收割机人脸表情和脑电信号进行融合。在实验中，受试者被要求观看旨在引起情绪反应的片段，然后报告他们的情绪状态。我们使用了两个情绪分析数据库使用生理信号（DEAP）和MAHNOB-HCI（MAHNOB-HCI）来评估我们的方法。此外，我们还进行了在线实验，使我们的方法更强大。我们的实验表明，我们的方法产生最先进的结果，在二进制效价/唤醒分类，基于DEAP和MAHNOB-HCI数据集。此外，对于在线实验，我们在融合后获得了69.75%的效价空间准确率和70.00%的唤醒空间准确率，每个都超过了性能最高的单一模态（效价空间69.28%和唤醒空间64.00%）。结果表明，面部表情和脑电信息相结合的情感识别弥补了其单一信息源的缺陷。这项工作的新奇如下。开始，我们结合面部表情和脑电来提高情绪识别的性能。此外，我们使用迁移学习技术来解决缺乏数据的问题，并实现更高的面部表情准确性。最后，除了实现广泛使用的基于枚举两个模型之间的不同权重的融合方法之外，我们还探索了一种新的融合方法，应用boosting技术。
Emotion recognition plays an essential role in human-computer interaction. Previous studies have investigated the use of facial expression and electroencephalogram (EEG) signals from single modal for emotion recognition separately, but few have paid attention to a fusion between them. In this paper, we adopted a multimodal emotion recognition framework by combining facial expression and EEG, based on a valence-arousal emotional model. For facial expression detection, we followed a transfer learning approach for multi-task convolutional neural network (CNN) architectures to detect the state of valence and arousal. For EEG detection, two learning targets (valence and arousal) were detected by different support vector machine (SVM) classifiers, separately. Finally, two decision-level fusion methods based on the enumerate weight rule or an adaptive boosting technique were used to combine facial expression and EEG. In the experiment, the subjects were instructed to watch clips designed to elicit an emotional response and then reported their emotional state. We used two emotion datasetsa Database for Emotion Analysis using Physiological Signals (DEAP) and MAHNOB-human computer interface (MAHNOB-HCI)to evaluate our method. In addition, we also performed an online experiment to make our method more robust. We experimentally demonstrated that our method produces state-of-the-art results in terms of binary valence/arousal classification, based on DEAP and MAHNOB-HCI data sets. Besides this, for the online experiment, we achieved 69.75% accuracy for the valence space and 70.00% accuracy for the arousal space after fusion, each of which has surpassed the highest performing single modality (69.28% for the valence space and 64.00% for the arousal space). The results suggest that the combination of facial expressions and EEG information for emotion recognition compensates for their defects as single information sources. The novelty of this work is as follows. To begin with, we combined facial expression and EEG to improve the performance of emotion recognition. Furthermore, we used transfer learning techniques to tackle the problem of lacking data and achieve higher accuracy for facial expression. Finally, in addition to implementing the widely used fusion method based on enumerating different weights between two models, we also explored a novel fusion method, applying boosting technique.