Collaborative Research DHB: Coordinated motion and facial expression in dyadic conversation

当人类交谈时，语义言语内容伴随着声音韵律(说话的重点和时机)、点头、眼球运动、眉毛抬起和微笑等嘴部表情。当一个人产生的动作预示着另一个人的对称动作时，可以观察到谈话者的动作和/或面部表情之间的协调：对称的形成。这种对称性的形成和随后的非语言行为中对称性的破坏之间的相互作用是交际过程中不可或缺的，也是人类社会交往动态的诊断。PIs提出了一个模型，在该模型中，来自听觉、视觉和本体感觉(对关节角度的感知)的低水平贡献被结合在一个镜像系统中，该系统通过形成和打破对话者的动作、面部表情和声音韵律之间的对称性来帮助情感和语义交流。在目前的项目中，天真的参与者将通过闭路电视系统与训练有素的实验室助理进行二元(一对一)对话，闭路电视系统显示计算机重建的实验室助理的头和脸。幼稚的参与者和实验室助理的动作、面部表情和发声都将被记录下来。天真的参与者可以获得的视觉和听觉刺激将被操纵，以提供关于头部运动、面部表情和声音韵律影响的强度和时机的具体假设检验。视觉操作将由照片逼真的重建化身头部(即计算机动画)提供，部分通过跟踪实验室助理的头部和面部来驱动，部分通过操纵化身运动和面部表情的定时和幅度来驱动。将构建一个用于头部运动和面部表情动力学的组合微分方程和计算模型，并在实时替代化身实现的实验室助手的头部动作或面部表情方面进行测试。该项目的更广泛影响分为三个主要领域：用于研究人类和社会动态的使能技术、用于精神病治疗的应用、用于人机接口设计和教育技术的应用。(1)这些实验将导致在社会互动中检验各种假设的方法的进步，其中研究问题涉及对感知的社会角色的操纵。(2)面部表情的自动化分析提供了对小群体、高压力环境中的社交互动的在线分析，在这些环境中，情绪调节至关重要，例如在住院精神治疗中心和心理治疗师与来访者的互动中。通过有效测量面部表情和相关的非语言行为，如头势和凝视，可以提高精神病学诊断、症状严重程度评估和治疗反应的可靠性、效度和实用性。(3)计算模型的成功结果可能会导致自动化计算机界面和辅导系统的发展，这些系统可以对学生困惑或理解的面部表情做出反应，从而指导更有效的教学和学习。

When humans converse, semantic verbal content is accompanied by vocal prosody (the emphasis and timing of speech), head nods, eye movements, eyebrow raises, and mouth expressions such as smiles. Coordination between conversants' movements and/or facial expressions can be observed when an action generated by one individual is predictive of a symmetric movement by another: symmetry formation. The interplay between such symmetry formation and subsequent symmetry breaking in nonverbal behavior is integral to the process of communication and is diagnostic of the dynamics of human social interaction. The PIs propose a model in which low level contributions from audition, vision, and proprioception (the perception of the angle of our joints) are combined in a mirror system that assists affective and semantic communication through the formation and breaking of symmetry between conversants' movements, facial expressions and vocal prosody. In the current project, naive participants will engage in dyadic (one-on-one) conversations with trained laboratory assistants over a closed-circuit video system that displays a computer reconstructed version of the lab assistant's head and face. Both the naive participant's and the lab assistant's motions, facial expressions, and vocalizations will be recorded. The visual and auditory stimuli available to the naive participant will be manipulated to provide specific hypothesis tests about the strength and timing of the effects of head movement, facial expression, and vocal prosody. The visual manipulation will be provided by a photorealistic reconstructed avatar head (i.e., a computer animation) driven partially by tracking the lab assistant's head and face, and partially from manipulation of timing and amplitude of the avatar's movement and facial expression. A combined differential equations and computational model for the dynamics of head movements and facial expression will be constructed and tested in real-time substitution for lab assistant's head motion or facial expression as realized by the avatar. The broader impact of this project falls into three main areas: enabling technology for the study of human and social dynamics, applications to the treatment of psychopathology, applications to human-computer interface design and educational technology. (1) These experiments will result in the advancement of methods for testing a wide variety of hypotheses in social interaction where the research question involves a manipulation of perceived social roles. (2) Automated analysis of facial expression provides on-line analysis of social interactions in small group, high stress settings in which emotion regulation is critical such as in residential psychiatric treatment centers and in psychotherapist-client interactions. The reliability, validity, and utility of psychiatric diagnosis, assessment of symptom severity, and response to treatment could be improved by efficient measurement of facial expression and related non-verbal behavior, such as head gesture and gaze. (3) Successful outcomes from the computational models may lead to the development of automated computer interfaces and tutoring systems that could respond to students' facial displays of confusion or understanding, and thereby guide more efficient instruction and learning.