CAREER: Multimodal and Multialgorithm Facial Activity Understanding by Audiovisual Information Fusion

职业：通过视听信息融合进行多模式和多算法面部活动理解

• 批准号: 1149787
• 负责人: Yan Tong
• 金额: $ 44.38万
• 依托单位: University South Carolina Research Foundation
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2012
• 资助国家: 美国
• 起止时间: 2012-03-01 至 2018-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1149787&HistoricalAwards=false
• 关键词: CAREER; Multimodal; Multialgorithm; Facial; Activity

This project develops a unified multimodal and multialgorithm fusion framework to recognize facial action units, which describe complex and rich facial behaviors. The information from voice is incorporated with visual observations to effectively improve facial activity understanding since voice and facial activity are intrinsically correlated. The developed framework systematically captures the inherent interactions between the visual and audio channels in a global context of human perception of facial behavior. Advanced machine learning techniques are developed to integrate these relationships together with uncertainties associated with various visual and audio measurements in the fusion framework to achieve a robust and accurate understanding of facial activity. It is these coordinated and consistent interactions that produce a meaningful facial display.The research work from this project fosters computer vision and machine learning technologies with applications across a wide range of fields varying from psychiatry to human-computer interaction. The new audiovisual emotional database constructed in this research facilitates benchmark evaluations and promotes new research directions, especially, in human behavior analysis. An integration of research and education promotes cutting-edge training on human-computer interactions to K-12, undergraduate, and graduate students, especially encourages the participation of women in engineering and computing.

该项目开发了一个统一的多模态和多算法融合框架来识别面部动作单元，这些动作单元描述了复杂而丰富的面部行为。来自语音的信息与视觉观察相结合，以有效地提高面部活动的理解，因为语音和面部活动是内在相关的。所开发的框架系统地捕捉在全球范围内的人类感知的面部行为的视觉和音频通道之间的固有的相互作用。先进的机器学习技术被开发来将这些关系与融合框架中与各种视觉和音频测量相关联的不确定性集成在一起，以实现对面部活动的鲁棒和准确的理解。正是这些协调一致的互动产生了有意义的面部显示。该项目的研究工作促进了计算机视觉和机器学习技术的发展，其应用范围广泛，从精神病学到人机交互。本研究所建构的新视听情绪资料库，有助于基准评估，并促进新的研究方向，特别是人类行为分析。研究和教育的整合促进了对K-12，本科生和研究生的人机交互的尖端培训，特别是鼓励妇女参与工程和计算。

项目成果

Improving Facial Expression Analysis using Histograms of Log-Transformed Nonnegative Sparse Representation with a Spatial Pyramid Structure

使用具有空间金字塔结构的对数变换非负稀疏表示的直方图改进面部表情分析

• DOI: 10.1109/fg.2013.6553774
• 发表时间: 2013
• 期刊: Proceedings of the International Conference on Automatic Face and Gesture Recognition
• 作者: Ping Liu, Shizhong Han

FACIAL GRID TRANSFORMATION: A NOVEL FACE REGISTRATION APPROACH FOR IMPROVING FACIAL ACTION UNIT RECOGNITION

面部网格变换：一种改进面部动作单元识别的新颖的面部注册方法

• DOI: 10.1109/icip.2014.7025283
• 发表时间: 2014
• 期刊: Proceedings - International Conference on Image Processing
• 作者: Han, Shizhong;Meng, Zibo;Liu, Ping;Tong, Yan
• 通讯作者: Tong, Yan

Feature Disentangling Machine - A Novel Approach of Feature Selection and Disentangling in Facial Expression Analysis

• DOI: 10.1007/978-3-319-10593-2_11
• 发表时间: 2014-09
• 作者: Ping Liu;Joey Tianyi Zhou;I. Tsang;Zibo Meng;Shizhong Han;Yan Tong