The origin of interindividual differences in audiovisual speech perception, combining neuroimaging, eye movements and behavior

言语是人类交流的最常见形式，从根本上说是多感官的：观察说话者的嘴使我们能够识别模糊的听觉信息。然而，最近的研究表明，在语音处理过程中，视觉信息的使用存在很大的个体差异。这很容易用著名的麦格克错觉来证明，在这种错觉中，两个不同的听觉和视觉音节的组合会导致第三个完全不同的音节的出现。 虽然有些受试者受到视觉语言的强烈影响，并且总是经历幻觉，但其他人从未这样做。最近，人们发现了这种变异性的两种可能解释。首先，经常感知McGurk效应的人更有可能注视说话者的嘴，McGurk频率和看嘴的时间之间存在显着相关性。第二，在神经元水平上，观察到感知McGurk效应的个体在上级颞沟（STS）中具有更大的活动，STS是介导多感觉整合和视听言语感知的大脑结构。这两个观察结果提出了一个明显的问题：眼球运动驱动STS活动吗？或者是相反？这是第一次，我们将测量视听语音感知（用McGurk效应测量），STS中的神经活动（用fMRI测量）和眼球运动（用红外线眼动跟踪测量），并使用结构方程模型来解开它们之间的因果关系。

Speech is the most common form of human communication and is fundamentally multisensory: observing the mouth of the talker allows us to identify otherwise ambiguous auditory information. However, recent discoveries indicate wide interindividual differences in the use of visual information during speech processing. This is easily demonstrated with the well known McGurk illusion, in which combining two different auditory and visual syllables causes the percept of a third, completely different, syllable. While some subjects are strongly influenced by the visual speech and always experience the illusion, others never do. Recently, two possible explanations for this variability were discovered. First, frequent perceivers of the McGurk effect were more likely to fixate the mouth of the talker, with a significant correlation between McGurk frequency and mouth looking time. Second, on a neuronal level, individual who perceived the McGurk effect were observed to have greater activity in the superior temporal sulcus (STS), a brain structure mediating multisensory integration and audiovisual speech perception. These two observations raise an obvious question: do eye movements drive STS activity? Or is it vice versa? For the first time, we will measure audiovisual speech perception (measured with the McGurk effect), neural activity in the STS (measured with fMRI) and eye movements (measured with infrared eye tracking) and use structural equation modeling to untangle the causal relationship between them.