A computer vision toolbox for computational analysis of nonverbal social communication

用于非语言社交交流计算分析的计算机视觉工具箱

• 批准号: 10579838
• 负责人: Birkan Tunc
• 金额: $ 82.25万
• 依托单位: CHILDREN'S HOSP OF PHILADELPHIA
• 依托单位国家: 美国
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-05-01 至 2025-02-28
• 项目状态: 未结题
• 来源: https://reporter.nih.gov/project-details/10579838
• 关键词: Adolescent; Age; Area; Autism Diagnosis; Behavior; Behavioral; Behavioral Research; Behavioral Sciences; Body measure procedure; Characteristics; Child; Classification; Clinical; Code; Communication; Community Health; Complex; Computational Technique; Computer Analysis; Computer Vision Systems; Cues; Data; Democracy; Development; Diagnostic; Dimensions; Disease; Educational Materials; Engineering; Face; Facial Expression; Genetic study; Goals; Grain; Grant; Human; Individual; Individual Differences; Intervention; Joints; Literature; Machine Learning; Measurement; Measures; Mental Health; Methods; Motion; Movement; Nature; Neurologic; Neurosciences; Nonverbal Communication; Oral cavity; Participant; Perception; Performance; Persons; Phenotype; Population Heterogeneity; Production; Public Health; Publishing; Reproducibility; Research; Research Personnel; Sampling; Science; Scientist; Sex Differences; Social Behavior; Social Development; Social Environment; Social Functioning; Social Interaction; Speech; System; Techniques; Teenagers; Time; Training; Translations; Validation; Work; analysis pipeline; autism spectrum disorder; automated algorithm; behavior measurement; behavioral health; clinical application; computerized tools; design; individual variation; interest; kinematics; machine learning method; non-verbal; novel; open source; sex; social communication; tool; usability; young adult

PROJECT SUMMARY We will develop novel computer vision tools to reliably and precisely measure nonverbal social communication through quantifying communicative facial and bodily expressions. Our tools will be designed and developed in order to maximize their usability by non-engineer behavioral scientists, filling the enormous gap between engineering advances and their clinical accessibility. Significance: Social interaction inherently relies on perception and production of coordinated face and body expressions. Indeed, atypical face and body movements are observed in many disorders, impacting social interaction and communication. Traditional systems for quantifying nonverbal communication (e.g., FACS, BAP) require extensive training and coding time. Their tedious coding requirements drastically limits their scalability and reproducibility. While an extensive literature exists on advanced computer vision and machine learning techniques for face and body analysis, there is no well-established method commonly used in mental health community to quantify production of facial and bodily expressions or efficiently capture individual differences in nonverbal communication in general. As a part of this proposal, we will develop a computer vision toolbox including tools that are both highly granular and highly scalable, to allow for measurement of complex social behavior in large and heterogeneous populations. Approach: Our team will develop tools that provide granular metrics of nonverbal social behavior, including localized face and body kinematics, characteristics of elicited expressions, and imitation performance. Our tools will facilitate measurement of social communication both within a person and between people, to allow for assessment of individual social communication cues as well as those that occur within bidirectional social contexts. Preliminary Data: We have developed and applied novel computer vision tools to assess: (1) diversity of mouth motion during conversational speech (effect size d=1.0 in differentiating young adults with and without autism during a brief natural conversation), (2) interpersonal facial coordination (91% accuracy in classifying autism diagnosis in young adults during a brief natural conversation, replicated in an independent child sample), and (3) body action imitation (85% accuracy in classifying autism diagnosis based on body imitation performance). As apart of current proposal, we will develop more generic methods that can be used in normative and clinical samples. Aims. In Aim 1, we will develop tools to automatically quantify fine-grained face movements and their coordination during facial expression production; in Aim 2, we will develop tools to quantify body joint kinematics and their coordination during bodily expression production; in Aim 3, we will demonstrate the tools’ ability to yield dimensional metrics using machine learning. Impact: Our approach is designed for fast and rigorous assessment of nonverbal social communication, providing a scalable solution to measure individual variability, within a dimensional and transdiagnostic framework.

项目概要 我们将开发新颖的计算机视觉工具来可靠、精确地测量非语言社交 通过量化交流面部和身体表情进行交流。我们的工具将是 设计和开发的目的是为了最大限度地提高非工程行为科学家的可用性，填补 工程进步与其临床可及性之间存在巨大差距。意义：社交互动 本质上依赖于协调的面部和身体表情的感知和产生。确实脸型不典型 在许多疾病中都观察到身体运动，影响社会互动和沟通。 用于量化非语言交流的传统系统（例如 FACS、BAP）需要大量的培训和 编码时间。它们繁琐的编码要求极大地限制了它们的可扩展性和可重复性。虽然 存在关于面部和身体的先进计算机视觉和机器学习技术的大量文献 分析表明，心理健康界没有普遍使用的成熟方法来量化产量 面部和身体表情或有效捕捉非语言交流中的个体差异 一般的。作为该提案的一部分，我们将开发一个计算机视觉工具箱，其中包括高度可用的工具 粒度和高度可扩展性，允许测量大规模和异构的复杂社会行为 人口。方法：我们的团队将开发提供非语言社交行为的精细指标的工具， 包括局部面部和身体运动学、引发的表情特征以及模仿表现。 我们的工具将有助于测量人体内和人与人之间的社交沟通，以允许 用于评估个人社交沟通线索以及双向社交中发生的线索 上下文。初步数据：我们开发并应用了新颖的计算机视觉工具来评估：（1）多样性 会话言语期间嘴部运动的影响（效应大小 d = 1.0 区分年轻人有和没有 简短的自然对话期间的自闭症），（2）人际面部协调（分类准确度为 91%） 在简短的自然对话中对年轻人进行自闭症诊断，并在独立的儿童样本中进行复制）， (3) 身体动作模仿（基于身体模仿对自闭症诊断进行分类的准确率达到 85%） 表现）。作为当前提案的一部分，我们将开发更多可用于规范性的通用方法 和临床样本。目标。在目标 1 中，我们将开发自动量化细粒度面部运动的工具 以及它们在面部表情产生过程中的协调；在目标2中，我们将开发量化身体关节的工具 身体表达产生过程中的运动学及其协调；在目标 3 中，我们将演示这些工具的 使用机器学习产生维度指标的能力。影响：我们的方法旨在快速且 对非语言社交沟通的严格评估，提供可扩展的解决方案来衡量个人 在维度和跨诊断框架内的可变性。