Computational quantification of emotion in faces and voice for neuropsychiatry

描述(由申请人提供)：有效的情感交流依赖于面部和声音变化等相互补充的非语言渠道的复杂相互作用。除了情绪表达外，通过复制情绪，情绪的模仿在理解情绪方面也起着重要作用。有效地利用这些面部-声音通道，并在许多神经精神障碍患者中识别其中的变化是受损的。量化这些缺陷将有助于推进基础和临床研究，最终导致提高诊断的准确性和治疗效果的评估。目前的方法主要依赖于临床评分，这些评分可能是主观的和视觉评分依赖的。可用于面部表情和声音分析的有限的自动化方法能够识别情绪，但在量化情绪程度方面相当不成功。这就产生了对客观的自动化情绪评估方法的需求，这些方法可以量化情绪，补充临床评级，并帮助诊断决策。这个项目试图通过开发和验证先进的自动化计算机化工具来解决这些问题，这些工具可以客观和可靠地量化多模式的影响处理。这项使用单一或组合的面部表情和声音视听方式对情绪表达和模仿进行的全面量化评估，将确定每个通道对情绪理解和患者对照组之间情感相关差异识别的影响，从而补充和增强现有的临床症状评分标准。我们生产的方法将易于使用，并可以促进大规模研究，衡量情感障碍和导致情感受损的障碍的影响变化。在目标1中，我们将开发和验证基于分类器的方法，用于基于自动时间动作单元轮廓的面部情感分析，用于量化语音存在下的面部情感表达和模仿。在目标2中，我们开发并验证了基于从声学信号中提取的频谱和韵律特征的情感分类器。这些将用表达和模仿的声音来量化情绪。最后，在目标3中，我们将创建一个基于视频的自动情绪表达量化系统，该系统融合了目标1和2中确定的面部和语音特征。设计的特定于人群的人脸-语音分类器集将最好地阐明患者在表情和模仿方面的控制差异。结果将与临床评分进行比较。我们期望在项目成功完成后，我们将拥有一套客观的面部和语音表情分析工具的集成集合，供神经精神科医生用于量化情绪障碍的程度和研究治疗效果。我们预计我们的方法将影响用于诊断精神分裂症的程序，可能还会影响情感障碍和自闭症谱系障碍的诊断。这些方法将是通用的，并可能进一步扩展到导致情绪表达障碍的其他神经精神病学或神经学疾病。

DESCRIPTION (provided by applicant): Effective communication of emotion relies on a complex interplay of complementary non-verbal channels involving facial and vocal changes. In addition to emotion expression, imitation of emotion also plays a major role in understanding emotion by replicating it Effective use of these face-voice channels and identification of the changes therein is impaired in many patients suffering from neuropsychiatric disorders Quantification of such deficits will help advance basic and clinical research, eventually leading to improved diagnostic accuracy and assessment of treatment effects. Current methodology relies most heavily on clinical ratings that may be subjective and visual rater dependent. The limited automated methods available for facial expression and voice analysis, are able to recognize the emotion, but have been fairly unsuccessful in quantifying the degree of emotion. This has created the need for objective automated methods of emotion evaluation that can quantify emotion, supplement clinical ratings and aid in diagnosis decisions. This project seeks to address these issues by developing and validating advanced automated computerized tools that can objectively and reliably quantify multimodal affect processing. This comprehensive quantified assessment of emotion expression and imitation using single or combined audio-visual modalities of facial expression and voice, will determine the impact of each channel on emotion understanding and on identification of affect related differences between patient-control groups, thereby complementing and augmenting current clinical symptom rating scales. The measures we produce will be easy to employ and could facilitate large-scale studies measuring impairment in affect and affect change across disorders that lead to impaired affect. In Aim 1 we will develop and validate classifier-based methods for facial affect analysis based on automated temporal action unit profiles, for quantifying facial emotion expression and imitation in the presence of speech. In Aim 2, we develop and validate emotion classifiers based on the spectral and prosodic features extracted from the acoustic signal. These will quantify emotion in expressed and imitated voice. Finally in Aim 3, we will create a video-based automated emotion expression quantification system that fuses facial and voice features identified in Aims 1 and 2. The population-specific set of face-voice classifiers designed will best elucidate patient-control differences in expression and imitation. Results will be compared to to clinical ratings. We expect that on successful completion of the project we will have an integrated collection of objective facial and speech expression analysis tools, usable by neuropsychiatrists to quantify the degree of emotion impairment and study treatment effects. We expect our methods to influence procedures used for diagnosing schizophrenia and perhaps affective disorders and autism spectrum disorders. The methods will be generic and could be further extended to other neuropsychiatric or neurological conditions that cause deficits in emotional expressiveness.