CAREER: Learning Nonverbal Signatures

非语言交际是我们日常生活中必不可少的组成部分。如果学生皱起眉头，就会告诉老师可能会感到困惑。在谈判中，歪着头可能会显示出怀疑。患者的微笑动态甚至可以预测他们未来的康复。这个项目将建立创新的技术，使计算机能够理解人类用户微妙的非语言行为。这个项目的主要新奇之处在于，它能够学习个体之间关于非语言行为如何表达的内在差异。同样的情感可以由不同的人表达得非常不同。非语言签名是指特定的人在与他人交谈时的视觉手势。通过识别这些非语言交流行为，如面部表情、头部手势和眼神交流，计算机和移动设备将能够更好地理解用户的情绪和情感状态。在心理健康治疗方面，开发的技术有可能改变医生用新的行为客观指标评估疾病的方式，并有可能预测患者的最终复发。对于虚拟角色的研究人员和动画师(例如，电影研究)，非语言签名将允许对最符合角色简介和个性的虚拟人非语言签名进行“点菜”选择。在教育和在线学习中，学生的非语言签名将有助于评估参与程度和感知与学习成功相关的情感状态。这个项目将倡导一种学习人类非语言行为的视觉表征的新范式：主要关注个人内部的可变性，其次是对群体结构的分析，并最终学习可在整个群体中推广的视觉表征。研究方法将受到非言语行为表达中的特质概念的充分研究。该项目将引入非语言签名的概念，这是一个人的非语言行为的低维计算表示，通过语言内容和情感上下文来实现。这个项目将解决四个基本的研究挑战：(1)个性化的非语言嵌入--学习非语言表现的个体变化的计算表征；(2)学习非语言签名--将非语言行为与语言和情感线索联系起来，以学习更有效的表征；(3)签名组合分析--从一组非语言签名中发现结构、原型和特质；(4)广义非语言表征--学习能够适应新个体的概括性非语言表征。这四项研究目标将由一项全面评价计划加以补充，该计划将包括四项中期评价和一项持续的总体评价。这项研究工作将为以人类为中心的新数据来源打开大门，在这些数据来源中，对非语言行为的准确解释是必不可少的。

Nonverbal communication is an essential component of our daily life. A quick eyebrow frown from a student will inform the teacher of possible confusion. A tilt of the head may show doubt during negotiation. A patient's smile dynamics can even predict their future recovery. This project will build innovative technologies to allow computers to understand subtle nonverbal behaviors of human users. The main novelty of this project will be in its capacity to learn the inherent variability between individuals on how nonverbal behaviors are expressed. The same emotion can be expressed very differently by different people. A nonverbal signature is how a specific person gestures visually when talking with other people. By recognizing these nonverbal communicative behaviors such as facial expressions, head gestures and eye contact, computers and mobile devices will be able to better understand the user's emotions and affective states. In mental health treatment, the developed technologies have the potential to change how doctors assess disorders with new behavioral objective measures and possibily to predict eventual relapse for the patient. For researchers and animators of virtual characters (e.g., movie studies), the nonverbal signatures will enable an "a la carte" selection of the virtual human nonverbal signature that best aligns with the character's profile and personality. In education and online learning, the student's nonverbal signature will help to assess engagement and perceive affective states related to success in learning. This project will advocate a novel paradigm for learning visual representations of human nonverbal behaviors: a major focus on intra-personal variability followed by analysis of group structures and eventual learning of visual representations generalizable across the population. The research methodology will be motivated by well-studied concepts of idiosyncrasy in nonverbal behavior expressions. The project will introduce the concept of nonverbal signatures which are low-dimensional computational representations of an individual's nonverbal behaviors contextualized by the verbal content and affective context. This project will address four fundamental research challenges: (1) personalized nonverbal embedding -- learning computational representations that summarize the individual variations in nonverbal appearance, (2) learning nonverbal signatures -- contextualizing the nonverbal behaviors with verbal and affective cues to learn more effective representations, (3) signature portfolio analysis -- discovering structure, prototypes and idiosyncrasy from a collection of nonverbal signatures, and (4) generalized nonverbal representations -- learning generalizable nonverbal representations able to adapt to new individuals. These four research aims will be complemented by a comprehensive evaluation plan to include four intermediate evaluations and a continuous overarching evaluation. This research effort will open the door to new sources of human-centric data where accurate interpretation of nonverbal behaviors is essential.