Multi-stage integration during co-speech gesture comprehension: a systematic investigation combining EEG and fMRI

手势是人际交流的重要组成部分。语音相关手势的处理需要不同感觉模态的整合，即，听觉和视觉输入的有意义的结合。然而，手势语音整合（GSI）不仅是多感官的性质，但在更高的语义水平。本文在前人对GSI的fMRI、EEG和同步EEG-fMRI研究的基础上，提出了MS-GSI模型（multi-stage model of gestree-speech integration），以解释GSI的神经生物学基础。MS-GSI模型假设GSI由从低阶视听感知和整合到高级语义匹配和修正过程的级联子阶段组成，并且这些阶段可以通过来自EEG和fMRI的不同且可分离的神经标记来表征。该模型为GSI的进一步整合和实证检验奠定了理论基础。然而，仍然存在一些挑战：首先，到目前为止，将EEG和fMRI的神经标记物直接统一到GSI的子阶段的努力有限。第二，目前的研究范式大多是从低层次的多感觉整合研究中获得的，对于语义层次的GSI的研究可能需要进一步的改进。第三，虽然手势的语义复杂性在GSI中起着关键作用，但在现有的文献中，它在很大程度上被忽视了。最后，尽管注意/预测对单一知觉和语言理解的影响普遍存在，但它们如何以自上而下的方式调节不同水平的GSI尚不清楚。为了解决这些问题，我们提出了目前的研究建议，包括三个工作包，使用EEG和同步EEG-fMRI来研究1）GSI中语义一致性和语义复杂性之间的相互作用，2）研究手势隐喻性的神经基础，一种主要来自手势的语义复杂性，一种新的设计，避免了现有多感觉范式的混淆，3）测试语义预测对较低水平GSI过程的自上而下的影响。每个工作包都针对GSI的特定子阶段，并将揭示对感官知觉和与社会交际功能相关的更高层次的感知，语义和建设性大脑过程之间相互作用的突破性新见解。总之，目前的研究项目将有助于我们更好地了解不同层次的加工与MS-GSI模型的神经生物学基础。这种理解不仅可以优化关于GSI的神经生物学模型，而且还可以为临床人群的翻译应用奠定坚实的基础，这些人群被证明是受损的GSI，这是一个高度社会相关的过程。

Gesture is a substantial component of interpersonal communication. The processing of speech-related gestures requires the integration of different sensory modalities, i.e., the meaningful combination of auditory and visual input. However, gesture-speech integration (GSI) is not only multisensory in nature, but at a higher semantic level. This unique feature has rendered GSI an emerging topic in cognitive neuroscience.Based on previous fMRI, EEG, and simultaneous EEG-fMRI studies on GSI, we proposed an MS-GSI model (the multi-stage model of gesture-speech integration), to account for the neurobiological basis of GSI. The MS-GSI model hypothesizes that GSI consists of cascaded sub-stages from lower-order audio-visual perception and integration, to higher-level semantic matching and revision processes, and that these stages can be characterized by distinct and dissociable neural markers from EEG and fMRI. The model forms a theoretical basis for further consolidation and empirical testing concerning GSI. However, there remain several challenges: firstly, there is limited effort so far to directly unify the neural markers from EEG and fMRI to the sub-stages of GSI. Secondly, the research paradigms, which are mostly adopted from studying of lower-level multisensory integration, may require further refinements for the investigation of semantic-level GSI. Thirdly, while semantic complexity of gesture plays a pivotal role during GSI, it is largely ignored in the existing literature. Lastly, despite the prevalence of top-down influence from attention/prediction to unisensory perception and language comprehension, it is not clear how they modulate different levels of GSI in a top-down manner.To resolve these issues, we propose the current research proposal with three work packages, using EEG and simultaneous EEG-fMRI to 1) investigate the interaction between semantic congruency and semantic complexity during GSI, 2) examine the neural basis of gestural metaphoricity, a semantic complexity that derives predominantly from gesture, with a new design that circumvents confounds from existing multisensory paradigms, and 3) testing the top-down influence of semantic prediction on lower-level GSI processes. Each work package targets at specific sub-stages of GSI, and will reveal ground breaking new insights into the interplay between sensory perception and higher-level perceptual, semantic, and constructive brain processes relevant for social communicative functioning. In sum, the current research project will help us to better understand the neurobiological basis of different levels of processing with the MS-GSI model. Such understanding may not only optimize the neurobiological models concerning GSI, but also form a solid basis for translational applications to clinic populations, which were shown to be impaired GSI, a process which is highly socially relevant.