Collaborative Research DHB: Coordinated motion and facial expression in dyadic conversation

当人类交谈时，语义语言内容伴随着有声韵律（说话的重点和时间）、点头、眼球运动、扬眉和嘴角表情（如微笑）。当一个人产生的动作可以预测另一个人的对称动作时，可以观察到熟悉者的动作和/或面部表情之间的协调：对称形成。在非语言行为中，这种对称形成和随后的对称打破之间的相互作用是沟通过程中不可或缺的一部分，也是对人类社会互动动态的诊断。pi提出了一个模型，在这个模型中，来自听觉、视觉和本体感觉（对关节角度的感知）的低水平贡献被结合在一个镜像系统中，通过在熟悉的动作、面部表情和声乐韵律之间形成和打破对称来帮助情感和语义交流。在当前的项目中，天真的参与者将通过闭路视频系统与训练有素的实验室助理进行一对一的对话，该系统显示了实验室助理头部和面部的计算机重建版本。天真的参与者和实验室助理的动作、面部表情和声音都将被记录下来。对幼稚参与者可用的视觉和听觉刺激进行处理，以提供关于头部运动、面部表情和声乐韵律影响的强度和时间的特定假设检验。视觉操作将由逼真的重建头像头部（即计算机动画）提供，部分通过跟踪实验室助理的头部和面部来驱动，部分通过操纵头像运动和面部表情的时间和幅度来驱动。将构建头部运动和面部表情动力学的组合微分方程和计算模型，并在虚拟化身实现的实验室助理的头部运动或面部表情的实时替代中进行测试。该项目的广泛影响主要集中在三个方面：人类和社会动态研究的使能技术，精神病理学治疗的应用，人机界面设计的应用和教育技术。(1)这些实验将导致测试社会互动中各种各样假设的方法的进步，其中研究问题涉及对感知社会角色的操纵。(2)面部表情的自动分析提供了在小群体、高压力环境下的社交互动的在线分析，在这些环境中，情绪调节是至关重要的，如在精神病治疗中心和心理治疗师与来访者的互动中。通过有效测量面部表情和相关的非语言行为，如头部手势和凝视，可以提高精神病诊断、症状严重程度评估和治疗反应的可靠性、有效性和实用性。(3)计算模型的成功结果可能导致自动化计算机界面和辅导系统的发展，这些系统可以对学生的困惑或理解的面部表情做出反应，从而指导更有效的教学和学习。

When humans converse, semantic verbal content is accompanied by vocal prosody (the emphasis and timing of speech), head nods, eye movements, eyebrow raises, and mouth expressions such as smiles. Coordination between conversants' movements and/or facial expressions can be observed when an action generated by one individual is predictive of a symmetric movement by another: symmetry formation. The interplay between such symmetry formation and subsequent symmetry breaking in nonverbal behavior is integral to the process of communication and is diagnostic of the dynamics of human social interaction. The PIs propose a model in which low level contributions from audition, vision, and proprioception (the perception of the angle of our joints) are combined in a mirror system that assists affective and semantic communication through the formation and breaking of symmetry between conversants' movements, facial expressions and vocal prosody. In the current project, naive participants will engage in dyadic (one-on-one) conversations with trained laboratory assistants over a closed-circuit video system that displays a computer reconstructed version of the lab assistant's head and face. Both the naive participant's and the lab assistant's motions, facial expressions, and vocalizations will be recorded. The visual and auditory stimuli available to the naive participant will be manipulated to provide specific hypothesis tests about the strength and timing of the effects of head movement, facial expression, and vocal prosody. The visual manipulation will be provided by a photorealistic reconstructed avatar head (i.e., a computer animation) driven partially by tracking the lab assistant's head and face, and partially from manipulation of timing and amplitude of the avatar's movement and facial expression. A combined differential equations and computational model for the dynamics of head movements and facial expression will be constructed and tested in real-time substitution for lab assistant's head motion or facial expression as realized by the avatar. The broader impact of this project falls into three main areas: enabling technology for the study of human and social dynamics, applications to the treatment of psychopathology, applications to human-computer interface design and educational technology. (1) These experiments will result in the advancement of methods for testing a wide variety of hypotheses in social interaction where the research question involves a manipulation of perceived social roles. (2) Automated analysis of facial expression provides on-line analysis of social interactions in small group, high stress settings in which emotion regulation is critical such as in residential psychiatric treatment centers and in psychotherapist-client interactions. The reliability, validity, and utility of psychiatric diagnosis, assessment of symptom severity, and response to treatment could be improved by efficient measurement of facial expression and related non-verbal behavior, such as head gesture and gaze. (3) Successful outcomes from the computational models may lead to the development of automated computer interfaces and tutoring systems that could respond to students' facial displays of confusion or understanding, and thereby guide more efficient instruction and learning.