The origin of interindividual differences in audiovisual speech perception, combining neuroimaging, eye movements and behavior

语言是人类交流最常见的形式，从根本上讲是多感官的：观察说话者的嘴部可以让我们识别出其他模棱两可的听觉信息。然而，最近的发现表明，在语音处理过程中，视觉信息的使用存在很大的个体差异。众所周知的McGurk错觉很容易证明这一点，其中结合两个不同的听觉和视觉音节会导致感知到第三个完全不同的音节。虽然一些受试者受到视觉语言的强烈影响，总是体验到幻觉，但其他人却从来没有。最近，人们发现了两种可能的解释。首先，频繁感知McGurk效应的人更有可能盯住说话人的嘴，McGurk频率和看嘴时间之间存在显著相关性。其次，在神经元水平上，观察到感知McGurk效应的个体在颞上沟（STS）中有更大的活动，这是一种调节多感觉整合和视听语言感知的大脑结构。这两个观察结果提出了一个明显的问题：眼球运动是否驱动STS活动？还是相反呢？我们将首次测量视听语音感知（用McGurk效应测量）、STS中的神经活动（用fMRI测量）和眼动（用红外眼动追踪测量），并使用结构方程建模来解开它们之间的因果关系。

Speech is the most common form of human communication and is fundamentally multisensory: observing the mouth of the talker allows us to identify otherwise ambiguous auditory information. However, recent discoveries indicate wide interindividual differences in the use of visual information during speech processing. This is easily demonstrated with the well known McGurk illusion, in which combining two different auditory and visual syllables causes the percept of a third, completely different, syllable. While some subjects are strongly influenced by the visual speech and always experience the illusion, others never do. Recently, two possible explanations for this variability were discovered. First, frequent perceivers of the McGurk effect were more likely to fixate the mouth of the talker, with a significant correlation between McGurk frequency and mouth looking time. Second, on a neuronal level, individual who perceived the McGurk effect were observed to have greater activity in the superior temporal sulcus (STS), a brain structure mediating multisensory integration and audiovisual speech perception. These two observations raise an obvious question: do eye movements drive STS activity? Or is it vice versa? For the first time, we will measure audiovisual speech perception (measured with the McGurk effect), neural activity in the STS (measured with fMRI) and eye movements (measured with infrared eye tracking) and use structural equation modeling to untangle the causal relationship between them.