Mapping Affect and Facial Dynamics during Dyadic Conversation

当我们与他人交谈时，面部表情、头部动作和非语言特征是我们情绪基调的重要线索。为了使这一过程有效，表达情感的线索必须不断变化，而且很难研究。缺乏测量和建模用于解释实时面部表情的多种线索的工具，限制了对对话中面部表情调节的科学研究。这一提议包括一些实验，旨在操纵特定的线索，以确定感知者在现场对话中对沟通者情绪状态的理解。这个项目的主要目标有两个。第一个目标是开发和完善工具（例如，计算机软件程序），使科学家能够更好地理解特定的面部和运动线索，使人们能够在现场对话中表达和理解交流的影响。第二个目标是完善允许这种情感交流测试的软件，使其成为一群希望研究和操纵实时互动本质的行为科学家的新工具。例如，这款软件将允许实验人员操纵正在进行现场相识对话的两个人的感知性别——这样，一个真正与另一个男人交谈的男人就能看到一个真实的、令人信服的女人的面部化身（她完美地模仿了实际男性互动伙伴的面部和头部动作）。在这个提议中开发和完善的软件将允许社会和认知心理学家以十年前无法想象的方式操纵，例如，人们互动伙伴的社会类别。这个项目的影响非常广泛。其中一个应用领域是测量和建模面部动态，也就是说，使用这项技术进一步研究实时面部表情。目前的项目将提供开源软件，可以免费分发用于研究目的。希望使用该软件的研究实验室还将获得免费的辅助材料（例如，计划、设备清单），以促进当前项目产生的技术的传播。第二个应用领域是跨文化交际的研究。这项新技术将允许对小团体中的社会互动和沟通进行复杂的研究，在这种情况下，情绪调节是至关重要的高压力环境（例如，警察或军事人员的跨文化互动）或谈判环境（例如，外交关系）。此外，这项技术还可以用于人与人之间的实时视频互动，既能保持交流的语气和内容，又能保持说话者的机密性。第三个应用领域是教育技术。虚拟学习环境的一个问题是很难发现学生的非语言线索——课堂老师经常用这些线索来评估学生的理解或困惑。这项工作可能会导致从视频捕捉中自动识别这些线索。这些信息可以用来改善虚拟学习环境，或者改进广播给学生的具体教学材料。

As we talk with others, facial expressions, head movements and nonverbal speech characteristics serve as crucial cues to our emotional tone. For this process to be effective, the cues to expressed affect must change constantly and are difficult to study. The lack of tools for measuring and modeling the multiple cues used to interpret live facial expressions has limited the scientific study of the regulation of facial expression in conversation. This proposal consists of experiments designed to manipulate the specific cues thought to determine perceivers' understanding of communicators' emotional states during live conversations. The primary goals of this project are twofold. The first goal is to develop and refine tools (e.g., computer software programs) that will allow scientists to better understand the specific facial and movement cues that allow people to express and understand communicated affect during live conversation. The second goal is to refine the software that allows such tests of affective communication, so that it serves as a new tool for a host of behavioral scientists who wish to study and manipulate the nature of live interactions. For example, this software will allow experimenters to manipulate the perceived gender of two people who are having a live getting-acquainted conversation -- so that a man who is actually talking to another man sees a realistic, convincing facial avatar of a woman (who perfectly mimics the facial and head movements of the actual male interaction partner). This software developed and refined in this proposal will thus allow social and cognitive psychologists to manipulate, for example, the social categories of people's interaction partners in ways not imagined a decade ago. The impact of this project is exceptionally wide ranging. One area of application is measuring and modeling facial dynamics -- that is, using this technology to further study live facial expression. The current project will provide open source software that can be distributed free for research purposes. Research labs who wish to use the software will also be provided with free supporting materials (e.g., plans, equipment lists) to facilitate the dissemination of the technology resulting from the current project. A second area of application is the study of intercultural communication. This new technology will allow sophisticated studies of social interactions and communication in small group, high stress settings in which emotional regulation is critical (e.g., intercultural interactions by police or military personnel) or in negotiation settings (e.g., diplomatic relations). Further the technology could be used to allow live video interactions between people that maintain the tone and content of communications while still maintaining speaker confidentiality. A third area of application is educational technology. A problem with virtual learning environments is the difficulty of detecting students' nonverbal cues -- cues that a live classroom teacher often uses to assess student understanding or confusion. The work may lead to automatic recognition of these cues from video capture. This information could then be used to improve the virtual learning environment or refine the specific teaching materials that are broadcast to students.