Computational quantification of emotion in faces and voice for neuropsychiatry

描述（由申请人提供）：有效的情感交流依赖于复杂的相互作用的互补非语言渠道，涉及面部和声音的变化。除了情绪表达，模仿情绪也通过复制它在理解情绪中起着重要作用。在许多患有神经精神障碍的患者中，这些面部-声音通道的有效使用和对其中变化的识别受到损害。量化这种缺陷将有助于推进基础和临床研究，最终导致提高诊断准确性和评估治疗效果。目前的方法最严重地依赖于临床评级，可能是主观的和视觉评级依赖。可用于面部表情和语音分析的有限的自动化方法能够识别情感，但在量化情感程度方面相当不成功。这就产生了对客观的自动化情绪评估方法的需求，这些方法可以量化情绪，补充临床评级并帮助诊断决策。该项目旨在通过开发和验证先进的自动化计算机化工具来解决这些问题，这些工具可以客观可靠地量化多模态情感处理。这种使用面部表情和声音的单一或组合的视听方式对情绪表达和模仿的综合量化评估，将确定每个通道对情绪理解的影响以及对患者-对照组之间的情感相关差异的识别，从而补充和增强当前的临床症状评定量表。我们产生的措施将很容易使用，并可以促进大规模的研究测量损害的影响和影响的变化，导致受损的影响障碍。在目标1中，我们将开发和验证基于分类器的方法，用于基于自动时间动作单元配置文件的面部情感分析，用于量化语音存在下的面部情感表达和模仿。在目标2中，我们开发和验证情感分类器的基础上提取的声学信号的频谱和韵律特征。这些将量化表达和模仿声音中的情感。最后，在目标3中，我们将创建一个基于视频的自动情感表达量化系统，该系统融合了目标1和目标2中确定的面部和语音特征。设计的人群特异性面部语音分类器将最好地阐明患者控制表达和模仿的差异。将结果与临床评级进行比较。我们希望在成功完成该项目后，我们将有一个客观的面部和言语表达分析工具的综合集合，可供神经精神科医生量化情绪障碍的程度和研究治疗效果。我们希望我们的方法能够影响用于诊断精神分裂症的程序，也许还有情感障碍和自闭症谱系障碍。这些方法将是通用的，并可进一步扩展到其他神经精神或神经疾病，导致情感表达的缺陷。 公共卫生相关性：该项目旨在通过开发先进的计算工具来量化面部表情和声音中的情感，这些工具将客观地缓解神经精神病学家用于研究疾病引起的情感产生障碍的主观情感评估方法所面临的挑战。这些经过充分验证的工具将应用于精神分裂症患者和对照组的视频数据集，以确定组间差异，研究疾病进展和治疗效果。

DESCRIPTION (provided by applicant): Effective communication of emotion relies on a complex interplay of complementary non-verbal channels involving facial and vocal changes. In addition to emotion expression, imitation of emotion also plays a major role in understanding emotion by replicating it Effective use of these face-voice channels and identification of the changes therein is impaired in many patients suffering from neuropsychiatric disorders Quantification of such deficits will help advance basic and clinical research, eventually leading to improved diagnostic accuracy and assessment of treatment effects. Current methodology relies most heavily on clinical ratings that may be subjective and visual rater dependent. The limited automated methods available for facial expression and voice analysis, are able to recognize the emotion, but have been fairly unsuccessful in quantifying the degree of emotion. This has created the need for objective automated methods of emotion evaluation that can quantify emotion, supplement clinical ratings and aid in diagnosis decisions. This project seeks to address these issues by developing and validating advanced automated computerized tools that can objectively and reliably quantify multimodal affect processing. This comprehensive quantified assessment of emotion expression and imitation using single or combined audio-visual modalities of facial expression and voice, will determine the impact of each channel on emotion understanding and on identification of affect related differences between patient-control groups, thereby complementing and augmenting current clinical symptom rating scales. The measures we produce will be easy to employ and could facilitate large-scale studies measuring impairment in affect and affect change across disorders that lead to impaired affect. In Aim 1 we will develop and validate classifier-based methods for facial affect analysis based on automated temporal action unit profiles, for quantifying facial emotion expression and imitation in the presence of speech. In Aim 2, we develop and validate emotion classifiers based on the spectral and prosodic features extracted from the acoustic signal. These will quantify emotion in expressed and imitated voice. Finally in Aim 3, we will create a video-based automated emotion expression quantification system that fuses facial and voice features identified in Aims 1 and 2. The population-specific set of face-voice classifiers designed will best elucidate patient-control differences in expression and imitation. Results will be compared to to clinical ratings. We expect that on successful completion of the project we will have an integrated collection of objective facial and speech expression analysis tools, usable by neuropsychiatrists to quantify the degree of emotion impairment and study treatment effects. We expect our methods to influence procedures used for diagnosing schizophrenia and perhaps affective disorders and autism spectrum disorders. The methods will be generic and could be further extended to other neuropsychiatric or neurological conditions that cause deficits in emotional expressiveness. PUBLIC HEALTH RELEVANCE: The project seeks to quantify emotion in facial expression and voice by developing advanced computational tools that will objectively alleviate the challenges faced by subjective methods of emotion evaluation used by neuropsychiatrists to study disease induced emotion production impairments. These well validated tools will be applied to video datasets of patients with schizophrenia and controls to determine group differences and study disease progression and treatment effects.