Multimodal assessment of dyadic interaction in disorders of social interaction.

自闭症谱系障碍（ASD）是一种典型的视觉和语言多模态交流障碍。观察工具，如自闭症诊断观察表（ADOS-2），通过与经验丰富的临床医生进行结构化的社会接触，对行为症状进行评估。基于该工具的诊断决策通常依赖于临床医生基于视觉交流行为（如眼神，面部表情，手势）及其与语言交流的整合的印象的定性临床评分，然而，缺乏定量指标。在人际交往中，语言和视觉的沟通渠道被嵌入到一个多模态结合的社会参考框架中。例如，共同注意来自使用指示手势（如指）配合面部表情和眼睛注视，在他人和物体的共同注意空间中导航。眼睛的注视和指向可以澄清话语所指的物体或人，例如，在谈论目前不可见的物体或人时，手势、头部运动和面部表情可以使空间和社会关系可视化。到目前为止，明确处理这种多模态视觉交流行为的相互作用和时间动态的系统研究很少，并且将受益于对二元交互的细粒度计算机评估。动作捕捉、移动眼动追踪和自动面部表情分析是这方面的成熟技术，并已被证明在自闭症等疾病的诊断评估方面具有潜力。然而，在以前的研究中缺少的是标准化评估过程中可用技术的多模态组合，从而产生丰富的、带注释的数据集。在本研究中，我们旨在提供典型和非典型社会行为的多模态评估，重点关注多种视觉和语言沟通渠道的整合及其与儿童社会互动障碍的关系。我们将对儿童和研究者持续互动过程中的特定行为事件进行注释，特别是与共同关注和互惠相关的行为。随后，我们将在自动提取方面的时间序列上使用机器学习（ML）方法（例如，对面部的扫视，面部表情，身体姿势动作捕捉）来训练模型以自动识别这些事件。此外，我们试图使用ML来分类典型和非典型行为，特别是与ASD相关的行为。同时，我们将生成一个丰富的数据集，即社会互动过程中的多模式通信语料库，这将允许在优先计划ViCom和更广泛的研究社区的背景下进行大量进一步的分析。

Autism Spectrum Disorder (ASD) is a prototypic disorder for the impairment of multimodal aspects of visual and verbal communication. Observational instruments such as the Autism Diagnostic Observation Schedule (ADOS-2) provide assessment of behavioral symptoms via a structured social encounter with an experienced clinician who performs several social interactive tasks with the individual. Diagnostic decisions based on this instrument typically rely on qualitative, clinical ratings regarding the clinician’s impression based on visual communicative behavior (such as eye-gaze, facial expressions, gestures) and their integration with verbal communication), however, quantitative indices are lacking.In human interaction, verbal and visual channels of communication are embedded in a social reference frame combining multimodal aspects. For example, joint attention emerges from using deictic hand gestures (e.g. pointing) in coordination with facial expression and eye gaze to navigate within a shared attentional space of other people and objects. Eye gaze and pointing can clarify which object or person an utterance refers to, e.g., gestures, head movement and facial expression may visualize spatial and social relationships when talking about objects or persons which are not currently visible. Systematic research that explicitly tackles the interplay and temporal dynamics of such multimodal visual communicative behavior is scarce, to date, and would benefit from fine-grained computerized assessment of dyadic interaction. Motion capture, mobile eye tracking and automatic facial expression analysis are established techniques in this respect, and have proved potential for the diagnostic assessment of disorders such as ASD. What is missing in previous research, however, is the multimodal combination of available techniques during a standardized assessment resulting in a rich, annotated dataset.In this proposal we aim at providing multimodal assessment of typical and atypical social behavior, focusing on the integration of multiple visual and verbal communication channels and their relation to disorders of social interaction in children. We will perform annotation of specific behavioral events during ongoing reciprocal interaction of a child and an investigator, in particular related to joint attention and reciprocity. Subsequently, we will use machine learning (ML) methods on time series of automatically extracted aspects (e.g. saccades towards faces, facial expression, body pose motion capture) to train models for the automatic identification of these events. Furthermore, we seek to use ML to classify typical and atypical behavior, in particular behavior related to ASD. At the same time, we will generate a rich dataset, i.e. a corpus of multimodal communication during social interaction which will allow for numerous further analyses in the context of the Priority Program ViCom and for the wider research community.