EAGER:Towards Adaptive and Robust Multimodal Emotion Recognition In-the-Wild

情感对人类生活至关重要。它们直接影响人类的感知和行为，并对人们的日常任务，如学习，社交和决策产生重大影响。自动情感识别已在人机交互、人机交互、多媒体检索、社交媒体分析和医疗保健等许多领域得到应用。情绪状态通过各种渠道表达，包括面部表情，语音韵律，口语和身体姿势。自动情感识别在现实世界中的应用是一个具有挑战性的任务。真实世界的情感涉及微妙的表达行为，不同渠道的不同程度的表达，以及不完美的条件，如背景噪音或音乐，照明不足，以及不受控制的头部姿势。这个早期概念的探索性研究项目旨在解决自发情感表达和不完美的音频和视频信号在野外的挑战，并开发一种新的多模态情感识别系统，用于现实世界的应用。这项研究将导致下一代情感计算在数据收集、算法设计和基准测试方面的进步。首先，将开发一个多模态数据集的自发情绪表达在野外。该数据集将包含各种具有挑战性的真实的生活环境中的自然自发情感数据，以及不同形式（音频和视频频道）的众包评级。一个彻底的基准分析使用这个数据集将进行研究如何不同的功能，模态和信号损伤有助于情绪识别系统的成功和失败。最后，新的多模态情感识别算法将设计使用自适应和鲁棒的多模态学习和融合。研究结果将通过数据集共享，出版物，讲座和开源代码提供，允许众多开发人员，研究人员和公司在现实世界的应用中改进和发展多模态情感识别。该项目还将为研究生和本科生，包括妇女和少数民族学生提供新的研究机会。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估。

Emotions are essential to human life. They directly influence human perception and behaviors, and have big impacts on people's daily tasks, such as learning, social interaction, and decision-making. Automatic emotion recognition has found applications in many domains such as human-computer interaction, human-robot interaction, multimedia retrieval, social media analysis, and healthcare. Emotional states are expressed through a variety of channels including facial expression, voice prosody, spoken words, and body gestures. Automatic emotion recognition in real-world applications is a challenging task. Real-world emotions involve subtle expressive behaviors, different degrees of expressiveness in different channels, and the imperfect conditions such as background noise or music, poor illumination, and uncontrolled head poses for example. This EArly-concept Grant for Exploratory Research project aims to address the challenges of spontaneous emotion expressions and imperfect audio and video signals in-the-wild, and develop a novel multimodal emotion recognition system for real-world applications. The research will lead to advances in data collection, algorithm design, and bench-marking for the next generation of affective computing.This project consists of several research components. First, a multimodal dataset of spontaneous emotion expressions in-the-wild will be developed. The dataset will contain natural spontaneous emotion data in various challenging real life environments, and crowd-sourced ratings in different modalities (audio and video channels). A thorough benchmark analysis using this dataset will be conducted tostudy how different features, modalities, and signal impairments contribute to the success and failure of emotion recognition systems. Finally, novel multimodal emotion recognition algorithms will be designed using adaptive and robust multimodal learning and fusion. The research findings will be made available through dataset sharing, publications, talks, and open-source codes, allowing a multitude of developers, researchers, and companies to improve and evolve multimodal emotion recognition in real-world applications. The project will also provide novel research opportunities for graduate and undergraduate students, including women and minority students.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.