Capturing Mismatch between Textual and Acoustic Emotion Expressions for Mood Identification in Bipolar Disorder
Capturing Mismatch between Textual and Acoustic Emotion Expressions for Mood Identification in Bipolar Disorder
复制标题
DOI：
10.21437/interspeech.2023-1990
复制
发表时间：
2023-08
期刊：
影响因子：
--
通讯作者：
Minxue Niu;Amrit Romana;Mimansa Jaiswal;M. McInnis;Emily Mower Provost
中科院分区：
文献类型：
--
作者：
Minxue Niu;Amrit Romana;Mimansa Jaiswal;M. McInnis;Emily Mower Provost
情感是一种复杂的行为现象，它通过语言、声音和面部表情等多种形式表达和感知。精神病学的研究表明，缺乏情感之间的一致性模态是情绪障碍的症状。在这项工作中，我们量化的情感表达通过语言和声学之间的不匹配，我们称之为情绪失配（EMM），作为一个中间步骤的情绪识别。我们使用从双相情感障碍（BP）患者中收集的纵向数据集，并表明症状性情绪发作与正常情绪相比显示出显着更多的EMM。我们提出了一个全自动的情绪识别流水线，自动语音转录，情感识别和EMM特征提取。我们发现，EMM功能，虽然尺寸较小，优于基于语言的基线，并始终提供改进时，结合语言和/或原始情感特征的情绪分类。
Emotion is a complex behavioral phenomenon, which is expressed and perceived through various modalities, such as language, vocal and facial expressions. Psychiatric research has suggested that the lack of emotional alignment between modalities is a symptom of emotion disorders. In this work, we quantify the mismatch between emotion expressed through language and acoustics, which we refer to as Emotional MisMatch (EMM), as an intermediate step for mood identification. We use a longitudinal dataset collected from people with Bipolar Disorder (BP) and show that symptomatic mood episodes show significantly more EMM, compared to euthymic moods. We propose a fully automatic mood identification pipeline with automatic speech transcription, emotion recognition, and EMM feature extraction. We find that EMM features, although smaller in size, outperform a language-based baseline, and consistently provide improvement when combined with language and/or raw emotion features on mood classification.