The origin of interindividual differences in audiovisual speech perception, combining neuroimaging, eye movements and behavior

语言是人类交流的最常见形式，从根本上讲是多感官的：观察说话者的嘴巴使我们能够识别原本含糊的听觉信息。然而，最近的发现表明，在语音处理过程中，视觉信息的使用存在很大的个体差异。这一点很容易用众所周知的麦格克错觉来证明，在这种错觉中，将两个不同的听觉和视觉音节结合起来，就会产生第三个完全不同的音节。虽然一些受试者受到视觉言语的强烈影响，并总是体验到错觉，但另一些受试者却从未经历过。最近，人们发现了这种差异的两种可能解释。首先，McGurk效应的频繁接受者更有可能注视说话者的嘴巴，McGurk频率和看嘴时间之间存在显著的相关性。其次，在神经元水平上，观察到感受到麦格克效应的个体在颞上沟(STS)有更大的活动，STS是一种调节多感觉整合和视听语音感知的大脑结构。这两个观察结果提出了一个显而易见的问题：眼球运动是否会驱动STS活动？或者反之亦然？我们将首次测量视听言语知觉(用McGurk效应测量)、STS的神经活动(用功能磁共振测量)和眼睛运动(用红外线眼球跟踪测量)，并使用结构方程模型来理清它们之间的因果关系。

Speech is the most common form of human communication and is fundamentally multisensory: observing the mouth of the talker allows us to identify otherwise ambiguous auditory information. However, recent discoveries indicate wide interindividual differences in the use of visual information during speech processing. This is easily demonstrated with the well known McGurk illusion, in which combining two different auditory and visual syllables causes the percept of a third, completely different, syllable. While some subjects are strongly influenced by the visual speech and always experience the illusion, others never do. Recently, two possible explanations for this variability were discovered. First, frequent perceivers of the McGurk effect were more likely to fixate the mouth of the talker, with a significant correlation between McGurk frequency and mouth looking time. Second, on a neuronal level, individual who perceived the McGurk effect were observed to have greater activity in the superior temporal sulcus (STS), a brain structure mediating multisensory integration and audiovisual speech perception. These two observations raise an obvious question: do eye movements drive STS activity? Or is it vice versa? For the first time, we will measure audiovisual speech perception (measured with the McGurk effect), neural activity in the STS (measured with fMRI) and eye movements (measured with infrared eye tracking) and use structural equation modeling to untangle the causal relationship between them.