A computer vision toolbox for computational analysis of nonverbal social communication

项目总结 我们将开发新的计算机视觉工具，以可靠和准确地测量非语言社交 通过量化可交流的面部和身体表情进行交流。我们的工具将是 设计和开发的目的是由非工程师行为科学家最大限度地提高它们的可用性，填补了 技术进步和临床可获得性之间的巨大差距。意义：社会互动 与生俱来地依赖于面部和身体协调表情的感知和产生。的确，非典型的面孔 在许多障碍中观察到身体运动，影响社会互动和交流。 用于量化非语言交流的传统系统(例如，FACS、BAP)需要大量的培训和 编码时间。它们繁琐的编码要求极大地限制了它们的可伸缩性和重现性。当一个 关于面部和身体的高级计算机视觉和机器学习技术存在着大量的文献 分析，在精神卫生领域还没有一种公认的方法来量化产量 或在非语言交流中有效地捕捉个体差异 将军。作为这项提议的一部分，我们将开发一个计算机视觉工具箱，其中包括高度 精细且高度可扩展，允许在大型和异类环境中测量复杂的社交行为 人口。方法：我们的团队将开发工具，为非语言社交行为提供细粒度的度量， 包括面部和身体的局部运动学，诱导表情的特征，以及模仿表演。 我们的工具将促进衡量一个人内部和人与人之间的社交交流，以允许 用于评估个人社交交流提示以及发生在双向社交网络中的提示 上下文。初步数据：我们开发和应用了新的计算机视觉工具来评估：(1)多样性 在会话语言中的嘴部运动(区分年轻人有无影响大小d=1.0 (2)人际面部协调(91%的分类准确率 在一次简短的自然对话中对年轻人进行自闭症诊断，并在独立的儿童样本中复制)， (3)身体动作模仿(基于身体模仿对自闭症诊断进行分类的准确率为85% 性能)。作为当前建议的一部分，我们将开发更通用的方法，这些方法可以用于规范 和临床样本。目标。在目标1中，我们将开发工具来自动量化细粒度的面部运动 以及它们在面部表情产生过程中的协调；在目标2中，我们将开发工具来量化身体关节 在身体表情产生过程中的运动学及其协调；在目标3中，我们将演示这些工具的 使用机器学习生成维度指标的能力。影响：我们的方法旨在实现快速和 严格评估非语言社交，提供可扩展的解决方案来衡量个人 变异性，在维度和跨诊断框架内。

PROJECT SUMMARY We will develop novel computer vision tools to reliably and precisely measure nonverbal social communication through quantifying communicative facial and bodily expressions. Our tools will be designed and developed in order to maximize their usability by non-engineer behavioral scientists, filling the enormous gap between engineering advances and their clinical accessibility. Significance: Social interaction inherently relies on perception and production of coordinated face and body expressions. Indeed, atypical face and body movements are observed in many disorders, impacting social interaction and communication. Traditional systems for quantifying nonverbal communication (e.g., FACS, BAP) require extensive training and coding time. Their tedious coding requirements drastically limits their scalability and reproducibility. While an extensive literature exists on advanced computer vision and machine learning techniques for face and body analysis, there is no well-established method commonly used in mental health community to quantify production of facial and bodily expressions or efficiently capture individual differences in nonverbal communication in general. As a part of this proposal, we will develop a computer vision toolbox including tools that are both highly granular and highly scalable, to allow for measurement of complex social behavior in large and heterogeneous populations. Approach: Our team will develop tools that provide granular metrics of nonverbal social behavior, including localized face and body kinematics, characteristics of elicited expressions, and imitation performance. Our tools will facilitate measurement of social communication both within a person and between people, to allow for assessment of individual social communication cues as well as those that occur within bidirectional social contexts. Preliminary Data: We have developed and applied novel computer vision tools to assess: (1) diversity of mouth motion during conversational speech (effect size d=1.0 in differentiating young adults with and without autism during a brief natural conversation), (2) interpersonal facial coordination (91% accuracy in classifying autism diagnosis in young adults during a brief natural conversation, replicated in an independent child sample), and (3) body action imitation (85% accuracy in classifying autism diagnosis based on body imitation performance). As apart of current proposal, we will develop more generic methods that can be used in normative and clinical samples. Aims. In Aim 1, we will develop tools to automatically quantify fine-grained face movements and their coordination during facial expression production; in Aim 2, we will develop tools to quantify body joint kinematics and their coordination during bodily expression production; in Aim 3, we will demonstrate the tools’ ability to yield dimensional metrics using machine learning. Impact: Our approach is designed for fast and rigorous assessment of nonverbal social communication, providing a scalable solution to measure individual variability, within a dimensional and transdiagnostic framework.