Predicting Autism and Social Functioning from Computer Vision Analyses of Motor Synchrony During Dyadic Interactions

通过计算机视觉对二元交互过程中运动同步的分析来预测自闭症和社交功能

• 批准号: 10540333
• 负责人: ROBERT Thomas SCHULTZ
• 金额: $ 64.6万
• 依托单位: CHILDREN'S HOSP OF PHILADELPHIA
• 依托单位国家: 美国
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-01-01 至 2024-11-30
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/10540333
• 关键词: Address; Adolescent; Adult; Agreement; Autism Diagnosis; Behavior; Behavioral; Biology; Categories; Child; Classification; Clinical; Clinical Trials; Code; Computer Vision Systems; Consumption; Data; Data Set; Diagnosis; Diagnostic; Dimensions; Disease; Face; Foundations; Genomics; Genotype; Goals; Grain; Grant; Human; Individual Differences; Limb structure; Machine Learning; Measurement; Measures; Mental disorders; Methodology; Methods; Modeling; Motion; Motor; Movement; Performance; Persons; Phenotype; Play; Property; Public Health; Publications; Questionnaires; Reporting; Reproducibility; Research; Role; Sampling; Scheme; Schools; Severities; Social Behavior; Social Functioning; Social Interaction; Specificity; Statistical Models; Symptoms; Techniques; Technology; Testing; Time; Training; Video Recording; Work; autism spectrum disorder; autistic children; behavioral phenotyping; biological development; brain based; computer framework; computerized; cost; design; diagnostic tool; digital; dyadic interaction; experimental study; impression; individual variation; informant; innovation; insight; lens; machine learning method; machine learning prediction; natural language; neuroimaging; novel; novel strategies; precision medicine; skills; social; social communication; social deficits; tool; wearable sensor technology; young adult

PROJECT SUMMARY Our innovative and fully automated approach to the analysis of social behavior addresses the pressing need for precise and scalable measurements of the autism spectrum disorder (ASD) phenotype. Using computer vision and machine learning methods, we have created a novel, quantitative method for fine-grained analysis of social interactions. Our approach directly measures interpersonal motor synchrony, a construct which we use as a lens for understanding the social interaction differences that are at the core of ASD. Significance: Genomics and neuroimaging methods continuously evolve, providing deeper insights into the biology of ASD. However, methods for measuring the outward manifestations of ASD have not changed substantially in decades. ASD is fundamentally a disorder of social interaction, but current clinical tools do not directly measure observable social interactions. Instead, they summarize global impressions of these interactions via informant report questionnaires or observational coding schemes that typically lack the behavioral granularity needed to robustly measure individual differences and changes across time (e.g., treatment related change). Inter-rater agreement on questionnaires is typically modest, while the alternative “deep phenotyping” by expert clinicians is a time- consuming and often cost-prohibitive burden to studies, especially when large samples are required (e.g., in genomics research). Approach: To resolve these problems, our team created a novel computational framework that leverages advances in markerless video motion capture, computer vision, and machine learning to directly capture dyadic social interactions. This allows us to capture all behaviors observable by expert clinicians but with exquisite digital precision and objectivity. Preliminary Data: We developed a fully automatic quantitative assessment of interpersonal social behavior focused on features of dyadic facial motor synchrony. When applied to videos of brief conversations between confederates and young adults with or without ASD, our assessment predicted diagnostic status with 91% accuracy – significantly better than highly trained clinical experts assessing the same video recordings. The set of predictive social motor synchrony features that we identified also correlated significantly with symptom severity in the ASD group, suggesting that it can be used for both diagnostic classification and evaluating individual differences (vital for advancing precision medicine goals). Importantly, our findings were reproducible across samples: the same features identified in our adult analysis also predicted diagnosis in a child sample with high accuracy. Aims. In Aim 1, we will test the specificity of our computer vision approach by expanding comparisons to include a mixed psychiatric disorder group; Aim 2 will test dyadic synchrony in other body movements, and Aim 3 will define associations between interpersonal motor synchrony and dimensional aspects of social communication that span diagnostic categories. Impact: Our approach is designed for fast and rigorous assessment of social communication, providing a scalable solution to diagnosing ASD diagnosis and measuring individual variability, within a transdiagnostic, precision medicine framework.

项目摘要 我们的创新和全自动化的方法来分析社会行为解决了紧迫的 需要精确和可扩展的自闭症谱系障碍（ASD）表型测量。使用 计算机视觉和机器学习方法，我们已经创造了一种新颖的，定量的方法， 社会互动分析。我们的方法直接测量人际运动同步性，这是一种结构， 我们用它作为一个透镜来理解ASD核心的社会互动差异。重要性： 基因组学和神经影像学方法不断发展，为ASD的生物学提供了更深入的见解。 然而，测量ASD外在表现的方法几十年来没有实质性变化。 ASD从根本上说是一种社会交往障碍，但目前的临床工具不能直接测量可观察到的 社交互动相反，他们通过线人报告总结了这些互动的全球印象 调查问卷或观察编码方案，通常缺乏稳健地 测量个体差异和随时间的变化（例如，治疗相关变化）。评核人间协议 问卷调查通常是温和的，而专家临床医生的替代“深度表型”是一个时间- 这对研究来说是消耗性的并且通常是成本过高的负担，特别是当需要大样本时（例如，在 基因组学研究）。方法：为了解决这些问题，我们的团队创建了一个新的计算框架 利用无标记视频运动捕捉、计算机视觉和机器学习的进步， 捕捉二元社会互动。这使我们能够捕获专家临床医生可观察到的所有行为， 精确的数字化和客观性。初步数据：我们开发了一种全自动定量 人际社会行为的评估侧重于双动面部运动同步性的特征。当应用 到有或没有ASD的联盟者和年轻人之间的简短对话的视频，我们的评估 预测诊断状态的准确率为91%-显著优于训练有素的临床专家评估 同样的录像我们发现的一组预测性社会运动同步特征也 与ASD组的症状严重程度显著相关，这表明它可以用于诊断 分类和评估个体差异（对于推进精准医疗目标至关重要）。重要的是， 我们的研究结果在不同的样本中是可重复的：在我们的成人分析中发现的相同特征也预测了 在儿童样本中的诊断具有高准确性。目标。在目标1中，我们将测试计算机视觉的特异性 通过扩大比较以包括混合精神障碍组的方法; Aim 2将测试二元 目标3将定义人际运动同步性与其他身体运动同步性之间的关联， 和社会沟通的维度方面，跨越诊断类别。影响：我们的方法是 专为快速和严格的社会沟通评估而设计，为诊断提供可扩展的解决方案 ASD诊断和测量个体变异性，在transdiagnosis，精准医学框架内。

项目成果

Gross motor impairment and its relation to social skills in autism spectrum disorder: A systematic review and two meta-analyses.

• DOI: 10.1037/bul0000358
• 发表时间: 2022-03
• 期刊: PSYCHOLOGICAL BULLETIN
• 影响因子: 22.4
• 作者: Wang, Leah A. L.;Petrulla, Victoria;Zampella, Casey J.;Waller, Rebecca;Schultz, Robert T.
• 通讯作者: Schultz, Robert T.

Computational Measurement of Motor Imitation and Imitative Learning Differences in Autism Spectrum Disorder: Computational Motor Imitation Measurement in ASD.

自闭症谱系障碍中运动模仿和模仿学习差异的计算测量：自闭症谱系障碍中的计算运动模仿测量。

• DOI: 10.1145/3461615.3485426
• 发表时间: 2021
• 期刊: ICMI '21 companion : companion publication of the 2021 International Conference on Multimodal Interaction : October 18th-22, 2021, Montreal, Canada. ICMI (Conference) (23rd : 2021 : Montreal, Quebec; Online)
• 作者: Zampella,CaseyJ;Sariyanidi,Evangelos;Hutchinson,AnneG;Bartley,GKeith;Schultz,RobertT;Tunç,Birkan
• 通讯作者: Tunç,Birkan

Discovering Synchronized Subsets of Sequences: A Large Scale Solution.

• DOI: 10.1109/cvpr42600.2020.00951
• 发表时间: 2020-06
• 期刊: Proceedings. IEEE Computer Society Conference on Computer Vision and Pattern Recognition
• 作者: Sariyanidi E;Zampella CJ;Bartley KG;Herrington JD;Satterthwaite TD;Schultz RT;Tunc B
• 通讯作者: Tunc B

Inequality-Constrained and Robust 3D Face Model Fitting

• DOI: 10.1007/978-3-030-58545-7_25
• 发表时间: 2020
• 期刊: Computer vision - ECCV ... : ... European Conference on Computer Vision : proceedings. European Conference on Computer Vision
• 作者: E. Sariyanidi;C. Zampella;R. Schultz;B. Tunç