CAREER: Multimodal and Multialgorithm Facial Activity Understanding by Audiovisual Information Fusion

该项目开发了一个统一的多模态和多算法融合框架来识别面部动作单元，这些动作单元描述了复杂而丰富的面部行为。来自语音的信息与视觉观察相结合，以有效地提高面部活动的理解，因为语音和面部活动是内在相关的。所开发的框架系统地捕捉在全球范围内的人类感知的面部行为的视觉和音频通道之间的固有的相互作用。先进的机器学习技术被开发来将这些关系与融合框架中与各种视觉和音频测量相关联的不确定性集成在一起，以实现对面部活动的鲁棒和准确的理解。正是这些协调一致的互动产生了有意义的面部显示。该项目的研究工作促进了计算机视觉和机器学习技术的发展，其应用范围广泛，从精神病学到人机交互。本研究所建构的新视听情绪资料库，有助于基准评估，并促进新的研究方向，特别是人类行为分析。研究和教育的整合促进了对K-12，本科生和研究生的人机交互的尖端培训，特别是鼓励妇女参与工程和计算。

This project develops a unified multimodal and multialgorithm fusion framework to recognize facial action units, which describe complex and rich facial behaviors. The information from voice is incorporated with visual observations to effectively improve facial activity understanding since voice and facial activity are intrinsically correlated. The developed framework systematically captures the inherent interactions between the visual and audio channels in a global context of human perception of facial behavior. Advanced machine learning techniques are developed to integrate these relationships together with uncertainties associated with various visual and audio measurements in the fusion framework to achieve a robust and accurate understanding of facial activity. It is these coordinated and consistent interactions that produce a meaningful facial display.The research work from this project fosters computer vision and machine learning technologies with applications across a wide range of fields varying from psychiatry to human-computer interaction. The new audiovisual emotional database constructed in this research facilitates benchmark evaluations and promotes new research directions, especially, in human behavior analysis. An integration of research and education promotes cutting-edge training on human-computer interactions to K-12, undergraduate, and graduate students, especially encourages the participation of women in engineering and computing.