Data-driven multidimensional modeling of nonverbal communication in typical and atypical development

典型和非典型发展中非语言交流的数据驱动多维建模

• 批准号: 10443715
• 负责人: Agata Rozga
• 金额: $ 32.76万
• 依托单位: GEORGIA INSTITUTE OF TECHNOLOGY
• 依托单位国家: 美国
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-08-01 至 2024-05-31
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/10443715
• 关键词: Accelerometer; Affect; Behavior; Behavioral; Child; Child Behavior; Child Development; Client; Code; Color; Computer Models; Data; Development; Developmental Delay Disorders; Diagnostic; Face; Future; Gestures; Goals; Grain; Hand; Head; Human; Intervention; Learning; Machine Learning; Measurement; Measures; Methodology; Methods; Modeling; Motor; Movement; Nonverbal Communication; Participant; Pattern; Posture; Production; Property; Research; Research Personnel; Risk; Series; Social Interaction; Statistical Models; Stream; Structure; Techniques; Testing; Time; Video Recording; Visit; Wrist; arm movement; autism spectrum disorder; autistic children; base; clinical care; common treatment; communication behavior; cost effective; data mining; diagnostic accuracy; efficacious treatment; gaze; improved; individualized medicine; innovative technologies; joint attention; language outcome; machine learning method; multimodality; novel; public health relevance; recruit; sensor; skills; social communication; tool; treatment planning

ABSTRACT This proposal will develop innovative technology for data-driven, multimodal characterization of nonverbal communication (NVC) in typical and atypical development. Prior research has provided qualitative descriptions of the development of children's use of gaze and gesture to regulate social interactions, but there are no objective, automated tools for measuring NVC behaviors, nor computational models to explain their coordination and timing in social interactions. This proposal will apply advanced probabilistic modeling techniques from machine learning and data mining to a rich corpus of children's behavior, including automated measures of children's posture, head pose, gaze direction, arm movements, and hand configurations derived from color and depth cameras and accelerometers. By automatically learning probabilistic latent variable models from movement data, we will obtain compact, data-driven descriptions of NVC and its coordination in children with autism, children with developmental delays without autism, and typically developing children (Aim 1). We will validate our models by demonstrating their ability to predict children's behavior, including diagnostic group and one-year language outcomes (Aim 2). We will test whether novel NVC patterns can be uncovered with bottom-up clustering of motor movement data (Aim 3). We predict our models will have greater explanatory and predictive power compared to current measures of NVC, which are typically human-coded behaviors that are descriptive, but rely on a-priori definitions of higher level behaviors. The models we develop will capture the fine-grained structure, coordination, and timing of NVC behaviors during social interactions, and thus allow us to characterize these behaviors with an unprecedented level of detail. Because interventions for young children with ASD target NVC skills, our automated measurement tools will provide clinicians with powerful new tools to assess the extent to which these treatments are efficacious. In addition, automated tools for dense measurement of fine-grained changes in NVC would enable clinicians to assess profiles of strengths and weaknesses for purposes of treatment planning, to dynamically tailor treatment to clients' changing abilities, and ultimately to accurately capture whether treatment is working. Finally, the measurement capabilities will provide researchers with a novel, cost-effective approach to analyze video recordings, at a scale that is not currently feasible due to a reliance on human coding.

摘要 这项提议将开发数据驱动的、非言语的多模式表征的创新技术 典型和非典型发展中的沟通(NVC)。先前的研究已经提供了定性的描述 关于儿童使用凝视和手势来调节社交互动的发展，但没有 客观、自动化的测量NVC行为的工具，也不是解释其行为的计算模型 社会互动中的协调和时间安排。该提案将应用高级概率建模 从机器学习和数据挖掘到丰富的儿童行为语料库，包括自动化 测量儿童的姿势、头部姿势、凝视方向、手臂动作和手形 从彩色和深度相机和加速计。通过自动学习概率潜变量 从运动数据建立模型，我们将获得NVC及其协调的紧凑、数据驱动的描述 自闭症儿童、没有自闭症的发育迟缓儿童，以及典型的发育中儿童(目标 1)。我们将通过展示它们预测儿童行为的能力来验证我们的模型，包括诊断 小组和一年的语言结果(目标2)。我们将测试是否可以发现新的NVC模式 使用自下而上的运动运动数据聚类(目标3)。我们预测我们的模型将有更大的 与NVC的当前衡量标准(通常是人类编码的)相比，解释能力和预测能力 行为是描述性的，但依赖于更高级别行为的先验定义。 我们开发的模型将捕获NVC行为的细粒度结构、协调和时间 在社会互动中，并因此允许我们用前所未有的水平来描述这些行为 细节。由于针对患有自闭症儿童的干预措施针对的是NVC技能，我们的自动化测量工具 将为临床医生提供强大的新工具，以评估这些治疗的有效程度。在……里面 此外，用于密集测量NVC细粒度变化的自动化工具将使临床医生能够 评估优点和缺点的概况，以便进行治疗计划，以动态调整 根据客户不断变化的能力进行治疗，并最终准确地捕获治疗是否有效。 最后，测量能力将为研究人员提供一种新的、经济有效的分析方法 视频记录，由于依赖人类编码，其规模目前不可行。

项目成果

Detection of eye contact with deep neural networks is as accurate as human experts.

• DOI: 10.1038/s41467-020-19712-x
• 发表时间: 2020-12-14
• 期刊: Nature communications
• 影响因子: 16.6
• 作者: Chong E;Clark-Whitney E;Southerland A;Stubbs E;Miller C;Ajodan EL;Silverman MR;Lord C;Rozga A;Jones RM;Rehg JM
• 通讯作者: Rehg JM