Effects of Physiological Signals in Different Types of Multimodal Sentiment Estimation
Effects of Physiological Signals in Different Types of Multimodal Sentiment Estimation
复制标题
DOI：
10.1109/taffc.2022.3155604
复制
发表时间：
2023-07
影响因子：
11.2
通讯作者：
Shun Katada;S. Okada;Kazunori Komatani
中科院分区：
文献类型：
--
作者：
Shun Katada;S. Okada;Kazunori Komatani
多模态情感分析是近年来的研究热点。然而，多模态情感分析的大多数研究仅考虑了人类可观察的信号，例如语言，音频和视觉信息，而这些信号与不可观察信号的多模态融合的贡献，即，生理信号，尚未得到全面的研究。在这项研究中，我们的目标是调查的生理信号在多模态情感分析的影响，通过评估所有的融合模型，在自然主义的人与智能体的交互设置不同类型的情感估计。我们的研究结果表明，生理特征是有效的单峰模型和语言表示与生理特征的融合提供了最好的结果，估计自我情感标签由用户自己注释。相比之下，具有视听特征的语言表示的张量融合对于在回归任务中估计由第三方注释的情感标签是有效的，其可以从人类可观察到的相应信号导出。对自我情绪估计结果的详细分析表明，不同的模态在情绪估计中起着不同的作用，并讨论了相应的含义。
Multimodal sentiment analysis has become a focus of research in recent years. However, most studies of multimodal sentiment analysis have considered only signals that are observable by humans, such as linguistic, audio and visual information, whereas the contribution of the multimodal fusion of such signals with unobservable signals, i.e., physiological signals, has not been comprehensively explored. In this study, we aim to investigate effects of physiological signals in multimodal sentiment analysis by evaluating all of the fusion models for different types of sentiment estimation in naturalistic human-agent interaction settings. Our results suggest that physiological features are effective in the unimodal model and that the fusion of linguistic representations with physiological features provides the best results for estimating self-sentiment labels as annotated by the users themselves. In contrast, the tensor fusion of linguistic representations with audiovisual features is effective for estimating sentiment labels as annotated by a third party in regression tasks, which can be derived from the corresponding signals that are observable by humans. A detailed analysis of the self-sentiment estimation results suggests that different modalities play different roles in sentiment estimation, and corresponding implications are discussed.