CI-ADDO-EN: Collaborative Research: 3D Dynamic Multimodal Spontaneous Emotion Corpus for Automated Facial Behavior and Emotion Analysis

情绪是个体心理状态的复杂心理生理体验。它影响到理性思考、学习、决策和精神运动能力的方方面面。情感建模和识别在人机交互、机器人学、人工智能以及先进的教育和学习技术等研究领域发挥着越来越重要的作用。然而，目前与情绪相关的研究由于缺乏大量的自发情绪数据语料库而受到阻碍。除了少数例外，情绪数据库在大小、传感器模式、标签和启发方法方面都是有限的。大多数人依赖于摆好姿势的情绪，这可能与情绪真正被触发的情境中发生的事情没有什么相似之处。在这个项目中，PI将通过使用不同模式的传感器开发来自大约200名不同种族和年龄的受试者的动态自发情绪和面部表情数据的多模式和多维语料库，以及标签和特征衍生品来解决这些限制。为此，他们将购买一台6摄像头广域3D动态成像系统，以捕捉超高分辨率的面部几何数据和视频纹理数据，这将使他们能够检查精细的结构变化以及自发表情的精确时间进程。视频数据将伴随着其他传感器形式，包括热传感器、音频传感器和生理传感器。红外热像仪将允许实时记录面部温度，而音频传感器将记录受试者和实验者的声音。生理传感器将测量皮肤电导率和相关的生理信号。将提供便利和简化数据集使用的工具和方法。整个数据集，包括元数据和相关软件，将被存储在公共储存库中，可用于计算机视觉、情感计算、人机交互和相关领域的研究。智力价值这项研究将涉及建立一个自发的多维和多模式情感和面部表情数据的语料库，其规模比目前存在的任何语料库都要大得多。为了从受试者身上激发自然和自发的情绪，PI将使用五种方法，包括物理体验、电影剪辑、冷加压、重温记忆任务和采访形式。该数据库将采用不同形式的传感器，包括高分辨率2D/3D摄像机、红外热像仪、音频传感器和生理传感器。视频数据将根据多个类别进行标记，包括AU标记和来自自我报告的情感标记以及天真观察者的感知判断。全面的情绪标记将包括维度方法(例如，价位、唤醒)、离散情绪(例如，喜悦、愤怒、微笑控制)、解剖学方法(例如，FAC)和副语言信号(例如，反向通道)。其他特征将从原始数据中提取，包括2D/3D面部特征点、头部姿势和音频参数。Broader Impact Project的成果将立即使计算机视觉和情感建模和识别方面的研究人员受益，因为数据库将允许他们训练和验证他们的面部表情和情感识别算法。新的语料库将促进从音频、视频、几何、热和物理响应中进行多模式融合的研究。它将有助于发展对涉及人类行为的机制的全面理解，并将允许加强人机交互(例如，通过情感敏感和社交智能界面)、机器人、人工智能和认知科学。这项工作还可能对心理学、生物统计学、医学/生命科学、执法、教育、诱骗和社会科学等其他不同领域的研究产生重大影响。

Emotion is the complex psycho-physiological experience of an individual's state of mind. It affects every aspect of rational thinking, learning, decision making, and psychomotor ability. Emotion modeling and recognition is playing an increasingly important role in many research areas, including human computer interaction, robotics, artificial intelligence, and advanced technologies for education and learning. Current emotion-related research, however, is impeded by a lack of a large spontaneous emotion data corpus. With few exceptions, emotion databases are limited in terms of size, sensor modalities, labeling, and elicitation methods. Most rely on posed emotions, which may bear little resemblance to what occurs in the contexts wherein the emotions are really triggered. In this project the PIs will address these limitations by developing a multimodal and multidimensional corpus of dynamic spontaneous emotion and facial expression data, with labels and feature derivatives, from approximately 200 subjects of different ethnicities and ages, using sensors of different modalities. To these ends, they will acquire a 6-camera wide-range 3D dynamic imaging system to capture ultra high-resolution facial geometric data and video texture data, which will allow them to examine the fine structure change as well as the precise time course for spontaneous expressions. Video data will be accompanied by other sensor modalities, including thermal, audio and physiological sensors. An IR thermal camera will allow real time recording of facial temperature, while an audio sensor will record the voices of both subject and experimenter. The physiological sensor will measure skin conductivity and related physiological signals. Tools and methods to facilitate and simplify use of the dataset will be provided. The entire dataset, including metadata and associated software, will be stored in a public depository and made available for research in computer vision, affective computing, human computer interaction, and related fields.Intellectual Merit This research will involve construction of a corpus of spontaneous multi-dimensional and multimodal emotion and facial expression data, which is significantly larger than any that currently exist. To elicit natural and spontaneous emotions from subjects, the PIs will employ five approaches using physical experience, film clips, cold pressor, relived memories tasks, and interview formats. The database will employ sensors of different modalities including high resolution 2D/3D video cameras, infrared thermal cameras, audio sensors, and physiological sensors. The video data will be labeled according to a number of categories, including AU labeling and emotion labeling from self-report and perceptual judgments of naïve observers. Comprehensive emotion labeling will include dimensional approaches (e.g., valence, arousal), discrete emotions (e.g., joy, anger, smile controls), anatomic methods (e.g., FACS), and paralinguistic signaling (e.g., back-channeling). Additional features will be derived from the raw data, including 2D/3D facial feature points, head pose, and audio parameters.Broader Impact Project outcomes will immediately benefit researchers in computer vision and emotion modeling and recognition, because the database will allow them to train and validate their facial expression and emotion recognition algorithms. The new corpus will facilitate the study of multimodal fusion from audio, video, geometric, thermal, and physical responses. It will contribute to the development of a comprehensive understanding of mechanisms involving human behavior, and will allow enhancements to human computer interaction (e.g., through emotion-sensitive and socially intelligent interfaces), robotics, artificial intelligence, and cognitive science. The work will likely also significantly impact research in diverse other fields such as psychology, biometrics, medicine/life science, law-enforcement, education, entrainment, and social science.