Utilizing Deep Learning Towards Multi-Modal Bio-Sensing and Vision-Based Affective Computing
Utilizing Deep Learning Towards Multi-Modal Bio-Sensing and Vision-Based Affective Computing
复制标题
DOI：
10.1109/taffc.2019.2916015
复制
发表时间：
2022-01-01
影响因子：
11.2
通讯作者：
Sejnowski, Terrence J.
中科院分区：
文献类型：
--
作者：
Siddharth;Jung, Tzyy-Ping;Sejnowski, Terrence J.
近年来，诸如脑电图（EEG）、心电图（ECG）等的生物感测信号的使用已经引起了对情感计算中的应用的兴趣。深度学习的并行趋势导致了性能的巨大飞跃，解决了各种基于视觉的研究问题，如对象检测。然而，深度学习的这些进展并没有充分转化为生物传感研究。这项工作将基于深度学习的新方法应用于四个公开的多模态情感数据集的各种生物传感和视频数据。对于每个数据集，我们首先单独评估每个模态获得的情感分类性能。然后，我们评估通过融合这些模式的功能所获得的性能。我们表明，我们的算法优于其他研究报告的结果DEAP和MAHNOB-HCI数据集上的情绪/效价/唤醒/喜欢分类，并为较新的AMIGOS和DREAMER数据集建立了基准。我们还通过组合数据集和使用迁移学习来评估我们的算法的性能，以表明所提出的方法克服了数据集之间的不一致性。因此，我们对来自120多名受试者和2,800次试验的多模态情感数据进行了全面分析。最后，利用卷积-去卷积网络，我们提出了一种新的技术，以确定显着的大脑区域对应于各种情感状态。
In recent years, the use of bio-sensing signals such as electroencephalogram (EEG), electrocardiogram(ECG), etc. have garnered interest towards applications in affective computing. The parallel trend of deep-learning has led to a huge leap in performance towards solving various vision-based research problems such as object detection. Yet, these advances in deep-learning have not adequately translated into bio-sensing research. This work applies novel deep-learning-based methods to various bio-sensing and video data of four publicly available multi-modal emotion datasets. For each dataset, we first individually evaluate the emotion-classification performance obtained by each modality. We then evaluate the performance obtained by fusing the features from these modalities. We show that our algorithms outperform the results reported by other studies for emotion/valence/arousal/liking classification on DEAP and MAHNOB-HCI datasets and set up benchmarks for the newer AMIGOS and DREAMER datasets. We also evaluate the performance of our algorithms by combining the datasets and by using transfer learning to show that the proposed method overcomes the inconsistencies between the datasets. Hence, we do a thorough analysis of multi-modal affective data from more than 120 subjects and 2,800 trials. Finally, utilizing a convolution-deconvolution network, we propose a new technique towards identifying salient brain regions corresponding to various affective states.