CI-ADDO-EN: Collaborative Research: 3D Dynamic Multimodal Spontaneous Emotion Corpus for Automated Facial Behavior and Emotion Analysis

情绪是个体心理状态的复杂的心理生理体验。 它影响理性思维、学习、决策和心理能力的各个方面。 情感建模和识别在许多研究领域中发挥着越来越重要的作用，包括人机交互，机器人，人工智能以及用于教育和学习的先进技术。 然而，目前情绪相关的研究受到缺乏大量自发情绪数据语料库的阻碍。 除了少数例外，情感数据库在大小、传感器模态、标记和启发方法方面受到限制。 大多数依赖于摆出的情绪，这可能与情绪真正被触发的背景下发生的事情几乎没有相似之处。 在这个项目中，PI将通过开发一个多模态和多维的动态自发情绪和面部表情数据语料库来解决这些限制，这些数据具有标签和特征衍生物，来自不同种族和年龄的大约200名受试者，使用不同模态的传感器。 为此，他们将获得一个6摄像头宽范围3D动态成像系统，以捕获超高分辨率的面部几何数据和视频纹理数据，这将使他们能够检查精细结构变化以及自发表达的精确时间过程。 视频数据将伴随着其他传感器形式，包括热，音频和生理传感器。 红外热感相机将允许真实的时间记录面部温度，而音频传感器将记录受试者和实验者的声音。生理传感器将测量皮肤电导率和相关的生理信号。 将提供便利和简化数据集使用的工具和方法。 整个数据集，包括元数据和相关软件，将被存储在一个公共的寄存器，并提供计算机视觉，情感计算，人机交互，以及相关领域的研究。智力优点这项研究将涉及建设一个语料库的自发多维和多模态的情感和面部表情数据，这是显着大于任何目前存在的。 为了从受试者中引出自然和自发的情感，PI将采用五种方法，包括物理体验、电影剪辑、冷加压、重温记忆任务和访谈形式。 该数据库将采用不同模态的传感器，包括高分辨率2D/3D摄像机、红外热成像摄像机、音频传感器和生理传感器。 视频数据将根据多个类别进行标记，包括来自天真观察者的自我报告和感知判断的Au标记和情感标记。 全面的情感标签将包括维度方法（例如，效价，唤醒），离散的情绪（例如，喜悦、愤怒、微笑控制），解剖学方法（例如，FACS）和语言信号（例如，反向信道）。 更广泛的影响项目的成果将立即使计算机视觉和情感建模和识别领域的研究人员受益，因为该数据库将允许他们训练和验证他们的面部表情和情感识别算法。 新的语料库将促进从音频，视频，几何，热和物理反应的多模态融合的研究。 它将有助于发展对涉及人类行为的机制的全面理解，并将允许增强人机交互（例如，通过情感敏感和社交智能接口），机器人技术，人工智能和认知科学。 这项工作也可能会对其他领域的研究产生重大影响，如心理学、生物识别、医学/生命科学、执法、教育、夹带和社会科学。

Emotion is the complex psycho-physiological experience of an individual's state of mind. It affects every aspect of rational thinking, learning, decision making, and psychomotor ability. Emotion modeling and recognition is playing an increasingly important role in many research areas, including human computer interaction, robotics, artificial intelligence, and advanced technologies for education and learning. Current emotion-related research, however, is impeded by a lack of a large spontaneous emotion data corpus. With few exceptions, emotion databases are limited in terms of size, sensor modalities, labeling, and elicitation methods. Most rely on posed emotions, which may bear little resemblance to what occurs in the contexts wherein the emotions are really triggered. In this project the PIs will address these limitations by developing a multimodal and multidimensional corpus of dynamic spontaneous emotion and facial expression data, with labels and feature derivatives, from approximately 200 subjects of different ethnicities and ages, using sensors of different modalities. To these ends, they will acquire a 6-camera wide-range 3D dynamic imaging system to capture ultra high-resolution facial geometric data and video texture data, which will allow them to examine the fine structure change as well as the precise time course for spontaneous expressions. Video data will be accompanied by other sensor modalities, including thermal, audio and physiological sensors. An IR thermal camera will allow real time recording of facial temperature, while an audio sensor will record the voices of both subject and experimenter. The physiological sensor will measure skin conductivity and related physiological signals. Tools and methods to facilitate and simplify use of the dataset will be provided. The entire dataset, including metadata and associated software, will be stored in a public depository and made available for research in computer vision, affective computing, human computer interaction, and related fields.Intellectual Merit This research will involve construction of a corpus of spontaneous multi-dimensional and multimodal emotion and facial expression data, which is significantly larger than any that currently exist. To elicit natural and spontaneous emotions from subjects, the PIs will employ five approaches using physical experience, film clips, cold pressor, relived memories tasks, and interview formats. The database will employ sensors of different modalities including high resolution 2D/3D video cameras, infrared thermal cameras, audio sensors, and physiological sensors. The video data will be labeled according to a number of categories, including AU labeling and emotion labeling from self-report and perceptual judgments of naïve observers. Comprehensive emotion labeling will include dimensional approaches (e.g., valence, arousal), discrete emotions (e.g., joy, anger, smile controls), anatomic methods (e.g., FACS), and paralinguistic signaling (e.g., back-channeling). Additional features will be derived from the raw data, including 2D/3D facial feature points, head pose, and audio parameters.Broader Impact Project outcomes will immediately benefit researchers in computer vision and emotion modeling and recognition, because the database will allow them to train and validate their facial expression and emotion recognition algorithms. The new corpus will facilitate the study of multimodal fusion from audio, video, geometric, thermal, and physical responses. It will contribute to the development of a comprehensive understanding of mechanisms involving human behavior, and will allow enhancements to human computer interaction (e.g., through emotion-sensitive and socially intelligent interfaces), robotics, artificial intelligence, and cognitive science. The work will likely also significantly impact research in diverse other fields such as psychology, biometrics, medicine/life science, law-enforcement, education, entrainment, and social science.