RI: Small: Collaborative Research: Exploring Audiovisual Emotion Perception using Data-Driven Computational Modeling

这个项目探索感知驱动的模型，人类视听情感使用统计分析，数据驱动的计算建模，和隐式传感。 情感是人类交流的基础和调节因素。 它被用于诊断许多心理健康状况，并在治疗干预中进行跟踪。 情绪感知的研究旨在确定描述“典型”情绪表达的感觉的模型--即，观察者/评估者对说话者情绪状态的归因。 这种感觉是个体整合呈现的多模态情感信息的方法的函数。 然而，多通道线索的相互作用的性质仍然是一个悬而未决的问题。 本研究将探讨情绪不一致性如何影响人的知觉判断，进而探讨多通道线索整合。 为了实现这一目标，本提案的研究目标是（1）识别和验证负责情绪感知的主要和次要视听线索，（2）创建一个数据驱动的模型来自动预测评估者的情绪感知，以及（3）使用隐式生理和身体姿势线索来预测评估者的状态。第一个研究目标是解决一个悬而未决的问题，即远端线索，说话者的交际目标的编码，如何相互作用，并导致特定的情绪状态的感觉。 新颖的技术将被用来识别情绪上突出的远端线索使用情绪一致和不一致的视听信息。 这种识别对情感分类算法的设计和情感代理的情感行为具有影响。 第二个研究重点解决了一个悬而未决的问题，即如何设计以人为中心的模型（而不是数据驱动的模型）用于情感分类任务。 该项目将研究新的动态结构对情感不一致信息建模的有效性。 这些新的结构将为以人为本的情感分类的发展提供见解，这些情感分类是由情感感知过程而不是仅仅基于数据波动而启发的。 第三个研究目标是解决视听情感评价任务对评价者内部状态的影响这一悬而未决的问题。 我们将评估评估者的注意力不集中的情绪评估任务的背景下。 可以准确预测评估者注意力不集中的模型在长期人机和人机交互平台中有应用。 从这个项目中获得的见解将有助于设计以情感为中心的算法，这些算法复制了人类解释和整合情感视听信号的过程。 它还将有助于为健康信息学应用程序创建情感界面，这将导致更有针对性的干预措施和治疗许多精神健康状况，包括精神分裂症，抑郁症和自闭症。

This project explores perception-driven models of human audio-visual emotion using statistical analyses, data-driven computational modeling, and implicit sensing. Emotion underlies and modulates human communication. It is used in the diagnosis of many mental health conditions and is tracked in therapeutic interventions. Research in emotion perception seeks to identify models that describe the felt sense of 'typical' emotion expression -- i.e., an observer/evaluator's attribution of the emotional state of the speaker. This felt sense is a function of the methods through which individuals integrate the presented multi-modal emotional information. However, the nature of the interaction of the multi-modal cues is still an open question. This project will investigate multi-modal cue integration by studying how emotional inconsistency affects human perceptual judgment. In pursuit of this goal, the research objectives of this proposal are (1) to identify and validate primary and secondary audio-visual cues responsible for emotion perception, (2) to create a data-driven model to automatically predict the emotion perception of an evaluator, and (3) to predict evaluator state using implicit physiological and body gesture cues. The first research objective addresses the open question of how distal cues, the encoding of a speaker's communicative goals, interact and result in the felt sense of specific emotion states. Novel techniques will be used to identify emotionally salient distal cues using emotionally consistent and inconsistent audio-visual information. This identification has implications in the design of emotion classification algorithms and the emotional behavior of affective agents. The second research thrust addresses the open question of how human-centered models (rather than data-driven models) can be designed for use in emotion classification tasks. The project will investigate the efficacy of novel dynamic structures to model emotionally inconsistent information. These new structures will provide insights into the development of human-centered emotion classification inspired by the emotion perception process, rather than solely on data fluctuations. The third research objective addresses the open question regarding the effect of audio-visual emotion evaluation tasks on the evaluator's internal state. We will assess evaluator inattention in the context of emotional evaluation tasks. Models that can accurately predict evaluator inattention have applications in long-term human-computer and human-robot interaction platforms. The insights gained from this project will facilitate the design of emotion-focused algorithms that replicate the process by which humans interpret and integrate emotional audiovisual signals. It will also aid in the creation of emotional interfaces for health informatics applications, which will lead to more specifically targeted interventions and treatments for many mental health conditions including schizophrenia, depression, and autism.