Mapping Affect and Facial Dynamics during Dyadic Conversation

当我们与他人交谈时，面部表情、头部动作和非言语特征对我们的情绪语气起着至关重要的作用。为了使这个过程有效，表达情感的线索必须不断变化，而且很难研究。缺乏工具来测量和建模用于解释实时面部表情的多个线索，这限制了对对话中面部表情调节的科学研究。这一建议包括一些实验，旨在操纵特定的思维线索，以确定接受者在现场对话中对交际者情绪状态的理解。这个项目的主要目标有两个。第一个目标是开发和改进工具(例如计算机软件程序)，使科学家能够更好地理解特定的面部和运动线索，使人们能够在现场对话中表达和理解交流的情感。第二个目标是改进允许对情感交流进行此类测试的软件，以便它成为一批希望研究和操纵现场互动性质的行为科学家的新工具。例如，这款软件将允许实验者操纵正在进行现场熟人对话的两个人的感知性别--这样，实际上正在与另一个男人交谈的男人就会看到一个逼真、令人信服的女人的面部化身(她完美地模仿了实际男性互动伙伴的面部和头部动作)。因此，在这项提议中开发和改进的软件将允许社会心理学家和认知心理学家以十年前无法想象的方式操纵人们互动伙伴的社交类别。这个项目的影响非常广泛。一个应用领域是测量和建模面部动力学--也就是使用这项技术来进一步研究活的面部表情。目前的项目将提供开源软件，可以免费分发用于研究目的。还将向希望使用该软件的研究实验室提供免费的辅助材料(例如，计划、设备清单)，以促进传播当前项目所产生的技术。第二个应用领域是跨文化交际的研究。这项新技术将允许对小群体、高压力环境中的社会互动和沟通进行复杂的研究，在这些环境中，情绪调节至关重要(例如，警察或军事人员的跨文化互动)或谈判环境(例如，外交关系)。此外，这项技术还可以用来实现人与人之间的实时视频互动，在保持通话语气和内容的同时，仍然保持发言者的机密性。第三个应用领域是教育技术。虚拟学习环境的一个问题是很难检测到学生的非语言暗示--现场教师经常使用这些暗示来评估学生的理解或困惑。这项工作可能会导致从视频捕获中自动识别这些线索。然后，这些信息可以用来改善虚拟学习环境或改进向学生广播的特定教材。

As we talk with others, facial expressions, head movements and nonverbal speech characteristics serve as crucial cues to our emotional tone. For this process to be effective, the cues to expressed affect must change constantly and are difficult to study. The lack of tools for measuring and modeling the multiple cues used to interpret live facial expressions has limited the scientific study of the regulation of facial expression in conversation. This proposal consists of experiments designed to manipulate the specific cues thought to determine perceivers' understanding of communicators' emotional states during live conversations. The primary goals of this project are twofold. The first goal is to develop and refine tools (e.g., computer software programs) that will allow scientists to better understand the specific facial and movement cues that allow people to express and understand communicated affect during live conversation. The second goal is to refine the software that allows such tests of affective communication, so that it serves as a new tool for a host of behavioral scientists who wish to study and manipulate the nature of live interactions. For example, this software will allow experimenters to manipulate the perceived gender of two people who are having a live getting-acquainted conversation -- so that a man who is actually talking to another man sees a realistic, convincing facial avatar of a woman (who perfectly mimics the facial and head movements of the actual male interaction partner). This software developed and refined in this proposal will thus allow social and cognitive psychologists to manipulate, for example, the social categories of people's interaction partners in ways not imagined a decade ago. The impact of this project is exceptionally wide ranging. One area of application is measuring and modeling facial dynamics -- that is, using this technology to further study live facial expression. The current project will provide open source software that can be distributed free for research purposes. Research labs who wish to use the software will also be provided with free supporting materials (e.g., plans, equipment lists) to facilitate the dissemination of the technology resulting from the current project. A second area of application is the study of intercultural communication. This new technology will allow sophisticated studies of social interactions and communication in small group, high stress settings in which emotional regulation is critical (e.g., intercultural interactions by police or military personnel) or in negotiation settings (e.g., diplomatic relations). Further the technology could be used to allow live video interactions between people that maintain the tone and content of communications while still maintaining speaker confidentiality. A third area of application is educational technology. A problem with virtual learning environments is the difficulty of detecting students' nonverbal cues -- cues that a live classroom teacher often uses to assess student understanding or confusion. The work may lead to automatic recognition of these cues from video capture. This information could then be used to improve the virtual learning environment or refine the specific teaching materials that are broadcast to students.