Multi-stage integration during co-speech gesture comprehension: a systematic investigation combining EEG and fMRI

手势是人际交往的重要组成部分。语言相关手势的处理需要不同感觉模式的整合，即听觉和视觉输入的有意义的结合。然而，手势语音整合（GSI）不仅在本质上是多感官的，而且在更高的语义层面上。这种独特的特征使GSI成为认知神经科学中的一个新兴课题。基于之前对GSI的fMRI、EEG和同时进行的EEG-fMRI研究，我们提出了一个MS-GSI模型（多阶段手势语音整合模型），以解释GSI的神经生物学基础。MS-GSI模型假设GSI由从低阶视听感知和整合到高阶语义匹配和修正过程的级联子阶段组成，并且这些阶段可以通过EEG和fMRI的不同且可分离的神经标记来表征。该模型为GSI的进一步巩固和实证检验奠定了理论基础。然而，仍然存在一些挑战：首先，目前在将EEG和fMRI的神经标记物直接统一到GSI的子阶段方面的努力有限。其次，在语义层面的GSI研究中，研究范式还有待进一步完善，目前多采用的是较低层次的多感觉整合研究范式。第三，虽然手势的语义复杂性在GSI中起着关键作用，但在现有文献中却被忽视了。最后，尽管普遍存在自上而下的影响，从注意/预测到感官感知和语言理解，但它们如何以自上而下的方式调节不同水平的GSI尚不清楚。为了解决这些问题，我们提出了目前的研究方案，包括三个工作包，使用EEG和同时使用EEG- fmri来1)研究GSI过程中语义一致性和语义复杂性之间的相互作用，2)研究手势隐喻的神经基础，手势隐喻是一种主要来自手势的语义复杂性，采用一种新的设计，避免了现有多感觉范式的混淆。3)检验语义预测对较低级GSI过程自上而下的影响。每个工作包都针对GSI的特定子阶段，并将揭示感官知觉与与社会交际功能相关的更高层次的感知、语义和建设性大脑过程之间相互作用的突破性新见解。总之，目前的研究项目将帮助我们更好地理解MS-GSI模型不同层次加工的神经生物学基础。这样的理解不仅可以优化关于GSI的神经生物学模型，而且可以为临床人群的转化应用奠定坚实的基础，这些人群被证明是GSI受损的，这是一个高度社会相关的过程。

Gesture is a substantial component of interpersonal communication. The processing of speech-related gestures requires the integration of different sensory modalities, i.e., the meaningful combination of auditory and visual input. However, gesture-speech integration (GSI) is not only multisensory in nature, but at a higher semantic level. This unique feature has rendered GSI an emerging topic in cognitive neuroscience.Based on previous fMRI, EEG, and simultaneous EEG-fMRI studies on GSI, we proposed an MS-GSI model (the multi-stage model of gesture-speech integration), to account for the neurobiological basis of GSI. The MS-GSI model hypothesizes that GSI consists of cascaded sub-stages from lower-order audio-visual perception and integration, to higher-level semantic matching and revision processes, and that these stages can be characterized by distinct and dissociable neural markers from EEG and fMRI. The model forms a theoretical basis for further consolidation and empirical testing concerning GSI. However, there remain several challenges: firstly, there is limited effort so far to directly unify the neural markers from EEG and fMRI to the sub-stages of GSI. Secondly, the research paradigms, which are mostly adopted from studying of lower-level multisensory integration, may require further refinements for the investigation of semantic-level GSI. Thirdly, while semantic complexity of gesture plays a pivotal role during GSI, it is largely ignored in the existing literature. Lastly, despite the prevalence of top-down influence from attention/prediction to unisensory perception and language comprehension, it is not clear how they modulate different levels of GSI in a top-down manner.To resolve these issues, we propose the current research proposal with three work packages, using EEG and simultaneous EEG-fMRI to 1) investigate the interaction between semantic congruency and semantic complexity during GSI, 2) examine the neural basis of gestural metaphoricity, a semantic complexity that derives predominantly from gesture, with a new design that circumvents confounds from existing multisensory paradigms, and 3) testing the top-down influence of semantic prediction on lower-level GSI processes. Each work package targets at specific sub-stages of GSI, and will reveal ground breaking new insights into the interplay between sensory perception and higher-level perceptual, semantic, and constructive brain processes relevant for social communicative functioning. In sum, the current research project will help us to better understand the neurobiological basis of different levels of processing with the MS-GSI model. Such understanding may not only optimize the neurobiological models concerning GSI, but also form a solid basis for translational applications to clinic populations, which were shown to be impaired GSI, a process which is highly socially relevant.