A computer vision toolbox for computational analysis of nonverbal social communication

用于非语言社交交流计算分析的计算机视觉工具箱

• 批准号: 10369663
• 负责人: Birkan Tunc
• 金额: $ 65.69万
• 依托单位: CHILDREN'S HOSP OF PHILADELPHIA
• 依托单位国家: 美国
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-05-01 至 2025-02-28
• 项目状态: 未结题
• 来源: https://reporter.nih.gov/project-details/10369663
• 关键词: Adolescent; Age; Area; Autism Diagnosis; Behavior; Behavioral; Behavioral Research; Behavioral Sciences; Characteristics; Child; Clinical; Code; Communication; Community Health; Complex; Computational Technique; Computer Analysis; Computer Vision Systems; Cues; Data; Development; Diagnostic; Dimensions; Disease; Educational Materials; Engineering; Face; Facial Expression; Genetic study; Goals; Gold; Grain; Grant; Human; Individual; Individual Differences; Intervention; Joints; Literature; Machine Learning; Measurement; Measures; Mental Health; Methods; Motion; Movement; Nature; Neurologic; Neurosciences; Nonverbal Communication; Oral cavity; Participant; Perception; Performance; Persons; Phenotype; Population Heterogeneity; Production; Public Health; Publishing; Reproducibility; Research; Research Personnel; Sampling; Science; Scientist; Sex Differences; Social Behavior; Social Development; Social Environment; Social Functioning; Social Interaction; Speech; System; Techniques; Teenagers; Time; Training; Translations; Validation; Work; analysis pipeline; autism spectrum disorder; automated algorithm; base; behavior measurement; behavioral health; clinical application; computerized tools; design; individual variation; interest; kinematics; machine learning method; novel; open source; sex; social; social communication; tool; usability; young adult

PROJECT SUMMARY We will develop novel computer vision tools to reliably and precisely measure nonverbal social communication through quantifying communicative facial and bodily expressions. Our tools will be designed and developed in order to maximize their usability by non-engineer behavioral scientists, filling the enormous gap between engineering advances and their clinical accessibility. Significance: Social interaction inherently relies on perception and production of coordinated face and body expressions. Indeed, atypical face and body movements are observed in many disorders, impacting social interaction and communication. Traditional systems for quantifying nonverbal communication (e.g., FACS, BAP) require extensive training and coding time. Their tedious coding requirements drastically limits their scalability and reproducibility. While an extensive literature exists on advanced computer vision and machine learning techniques for face and body analysis, there is no well-established method commonly used in mental health community to quantify production of facial and bodily expressions or efficiently capture individual differences in nonverbal communication in general. As a part of this proposal, we will develop a computer vision toolbox including tools that are both highly granular and highly scalable, to allow for measurement of complex social behavior in large and heterogeneous populations. Approach: Our team will develop tools that provide granular metrics of nonverbal social behavior, including localized face and body kinematics, characteristics of elicited expressions, and imitation performance. Our tools will facilitate measurement of social communication both within a person and between people, to allow for assessment of individual social communication cues as well as those that occur within bidirectional social contexts. Preliminary Data: We have developed and applied novel computer vision tools to assess: (1) diversity of mouth motion during conversational speech (effect size d=1.0 in differentiating young adults with and without autism during a brief natural conversation), (2) interpersonal facial coordination (91% accuracy in classifying autism diagnosis in young adults during a brief natural conversation, replicated in an independent child sample), and (3) body action imitation (85% accuracy in classifying autism diagnosis based on body imitation performance). As apart of current proposal, we will develop more generic methods that can be used in normative and clinical samples. Aims. In Aim 1, we will develop tools to automatically quantify fine-grained face movements and their coordination during facial expression production; in Aim 2, we will develop tools to quantify body joint kinematics and their coordination during bodily expression production; in Aim 3, we will demonstrate the tools’ ability to yield dimensional metrics using machine learning. Impact: Our approach is designed for fast and rigorous assessment of nonverbal social communication, providing a scalable solution to measure individual variability, within a dimensional and transdiagnostic framework.

项目总结