CI-ADDO-EN: Collaborative Research: 3D Dynamic Multimodal Spontaneous Emotion Corpus for Automated Facial Behavior and Emotion Analysis

CI-ADDO-EN：协作研究：用于自动面部行为和情绪分析的 3D 动态多模态自发情绪语料库

• 批准号: 1205664
• 负责人: Lijun Yin
• 金额: $ 30.68万
• 依托单位: SUNY at Binghamton
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2012
• 资助国家: 美国
• 起止时间: 2012-09-01 至 2016-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1205664&HistoricalAwards=false
• 关键词: CI; ADDO; EN; Collaborative; Research

Emotion is the complex psycho-physiological experience of an individual's state of mind. It affects every aspect of rational thinking, learning, decision making, and psychomotor ability. Emotion modeling and recognition is playing an increasingly important role in many research areas, including human computer interaction, robotics, artificial intelligence, and advanced technologies for education and learning. Current emotion-related research, however, is impeded by a lack of a large spontaneous emotion data corpus. With few exceptions, emotion databases are limited in terms of size, sensor modalities, labeling, and elicitation methods. Most rely on posed emotions, which may bear little resemblance to what occurs in the contexts wherein the emotions are really triggered. In this project the PIs will address these limitations by developing a multimodal and multidimensional corpus of dynamic spontaneous emotion and facial expression data, with labels and feature derivatives, from approximately 200 subjects of different ethnicities and ages, using sensors of different modalities. To these ends, they will acquire a 6-camera wide-range 3D dynamic imaging system to capture ultra high-resolution facial geometric data and video texture data, which will allow them to examine the fine structure change as well as the precise time course for spontaneous expressions. Video data will be accompanied by other sensor modalities, including thermal, audio and physiological sensors. An IR thermal camera will allow real time recording of facial temperature, while an audio sensor will record the voices of both subject and experimenter. The physiological sensor will measure skin conductivity and related physiological signals. Tools and methods to facilitate and simplify use of the dataset will be provided. The entire dataset, including metadata and associated software, will be stored in a public depository and made available for research in computer vision, affective computing, human computer interaction, and related fields.Intellectual Merit This research will involve construction of a corpus of spontaneous multi-dimensional and multimodal emotion and facial expression data, which is significantly larger than any that currently exist. To elicit natural and spontaneous emotions from subjects, the PIs will employ five approaches using physical experience, film clips, cold pressor, relived memories tasks, and interview formats. The database will employ sensors of different modalities including high resolution 2D/3D video cameras, infrared thermal cameras, audio sensors, and physiological sensors. The video data will be labeled according to a number of categories, including AU labeling and emotion labeling from self-report and perceptual judgments of naïve observers. Comprehensive emotion labeling will include dimensional approaches (e.g., valence, arousal), discrete emotions (e.g., joy, anger, smile controls), anatomic methods (e.g., FACS), and paralinguistic signaling (e.g., back-channeling). Additional features will be derived from the raw data, including 2D/3D facial feature points, head pose, and audio parameters.Broader Impact Project outcomes will immediately benefit researchers in computer vision and emotion modeling and recognition, because the database will allow them to train and validate their facial expression and emotion recognition algorithms. The new corpus will facilitate the study of multimodal fusion from audio, video, geometric, thermal, and physical responses. It will contribute to the development of a comprehensive understanding of mechanisms involving human behavior, and will allow enhancements to human computer interaction (e.g., through emotion-sensitive and socially intelligent interfaces), robotics, artificial intelligence, and cognitive science. The work will likely also significantly impact research in diverse other fields such as psychology, biometrics, medicine/life science, law-enforcement, education, entrainment, and social science.

情感是个体心理状态的复杂心理生理体验。它影响着理性思维、学习、决策和精神运动能力的各个方面。情感建模和识别在人机交互、机器人技术、人工智能以及先进的教育和学习技术等许多研究领域发挥着越来越重要的作用。然而，目前的情绪相关研究由于缺乏大量的自发情绪数据语料库而受到阻碍。除了少数例外，情感数据库在大小、传感器模式、标签和启发方法方面都是有限的。大多数人依赖于摆姿势的情绪，这与真正触发情绪的情境几乎没有相似之处。在这个项目中，pi将通过开发一个多模态和多维的动态自发情绪和面部表情数据语料库，使用不同模态的传感器，使用标签和特征衍生，来自大约200个不同种族和年龄的受试者。为此，他们将获得一个6摄像头宽范围3D动态成像系统，以捕获超高分辨率的面部几何数据和视频纹理数据，这将使他们能够检查细微的结构变化以及自发表情的精确时间过程。视频数据将伴随着其他传感器模式，包括热、音频和生理传感器。红外热像仪可以实时记录面部温度，而音频传感器可以记录受试者和实验者的声音。生理传感器将测量皮肤电导率和相关的生理信号。将提供工具和方法来促进和简化数据集的使用。整个数据集，包括元数据和相关软件，将存储在一个公共存储库中，供计算机视觉、情感计算、人机交互和相关领域的研究使用。本研究将涉及构建一个自发的多维、多模态情绪和面部表情数据语料库，该语料库比现有的任何语料库都要大得多。为了从被试中引出自然和自发的情绪，pi将采用五种方法，包括物理体验、电影剪辑、冷压力、重温记忆任务和访谈形式。该数据库将采用不同模式的传感器，包括高分辨率2D/3D摄像机、红外热像仪、音频传感器和生理传感器。视频数据将根据多个类别进行标记，包括AU标记和来自naïve观察者自我报告和感知判断的情感标记。综合情绪标签将包括维度方法（如效价、唤醒）、离散情绪（如喜悦、愤怒、微笑控制）、解剖学方法（如FACS）和副语言信号（如反向通道）。其他特征将从原始数据中导出，包括2D/3D面部特征点、头部姿势和音频参数。项目成果将立即使计算机视觉、情感建模和识别方面的研究人员受益，因为数据库将允许他们训练和验证他们的面部表情和情感识别算法。新的语料库将促进从音频、视频、几何、热和物理响应的多模态融合的研究。它将有助于对涉及人类行为的机制的全面理解的发展，并将允许增强人机交互（例如，通过情感敏感和社会智能接口）、机器人、人工智能和认知科学。这项工作可能还会对其他不同领域的研究产生重大影响，如心理学、生物识别学、医学/生命科学、执法、教育、娱乐和社会科学。