Neural substrates of optimal multisensory integration

项目摘要 当面对面交流时，人类通过两种感官方式接收信息：视觉信息 来自说话者的面部;以及来自说话者的声音的听觉信息。大流行病带来了 视听演讲的重要性：戴面具掩盖了说话者的脸， 声音，阻碍沟通的双重打击。视频会议软件的普及 这证明了人们对说话者的脸和声音的重视。虽然 视听言语感知是非常重要的，我们知道的很少，这独特的神经机制 人类的能力。我们将使用人类神经科学中最强大的技术来弥补这一知识差距： 计算建模;行为研究;超高场（7特斯拉）功能性MRI（fMRI）;以及 检查癫痫病人谁有电极植入他们的大脑治疗药物 难治性癫痫，一种被称为颅内脑电图（iEEG）的技术。解剖 该建议的重点是后上级颞沟/回（STS/G），自1989年以来就被人们所知。 韦尼克是重要的言语知觉。我们建议的行为焦点是新发现， 一种经典的视听言语错觉，被称为麦格克效应，可以产生戏剧性的，持久的变化， 在纯语音感知中，把ba变成da。这种现象，称为融合诱导 重新校准（FIR），提供了一个工具，以推进计算和神经研究的语音感知。第一 aim将开发言语感知的计算模型，并根据行为数据进行测试。不同型号 将适合于暴露于McGurk效应之前、期间和之后的言语感知数据。模型将 使用被保留的行为数据进行比较。由于模型实例化了不同的理论结构， 模型比较将确定哪些解释性结构是必不可少的。这些结果将提供一个坚实的 为未来的研究奠定理论基础，包括目标2和3中的研究： 不合理的建设很可能是徒劳的。第二个目标是通过以下透镜来考察言语知觉 STS/G的活动模式用7特斯拉的功能磁共振成像测量。我们希望在STS/G中观察到可靠的变化 暴露于McGurk效应前后的反应模式，反映了言语的修改 表示（相反，在仅由声学特征驱动的皮层区域中，McGurk暴露不应 改变fMRI反应模式。）第三个目标将使用iEEG记录宽带高频活动 (BHA)从STS/G神经元的小群体具有高时间分辨率。对听觉的反应- 只有McGurk语言的组成部分（而不是控制语言）被预测显示出持续下降后， 连续块的视听麦格克曝光，在锁步与知觉发展的FIR。

Project Summary When communicating face-to-face, humans receive information in two sensory modalities: visual information from the face of the talker; and auditory information from the voice of the talker. The pandemic has brought into sharp focus the importance of audiovisual speech: mask wearing obscures the talker's face while muffling the voice, a double whammy that hinders communication. The popularity of video conferencing software attests to the importance placed by people on seeing a talker's face as well as hearing their voice. Although audiovisual speech perception is very important, we know little about the neural mechanisms for this uniquely human ability. We will remedy this gap in knowledge using most powerful techniques in human neuroscience: computational modeling; behavioral studies; ultra-high-field (7 tesla) functional MRI (fMRI); and the examination of epilepsy patients who have electrodes implanted in their brain for the treatment of medically intractable epilepsy, a technique referred to as intracranial electroencephalography (iEEG). The anatomical focus of the proposal is the posterior superior temporal sulcus/gyrus (STS/G), known since the time of Wernicke to be important for speech perception. The behavioral focus of our proposal is the new discovery that a classic audiovisual speech illusion, known as the McGurk effect, can produce dramatic, long-lasting changes in auditory-only speech perception, turning a ba into a da. This phenomenon, termed fusion-induced recalibration (FIR), provides a tool to advance computational and neural studies of speech perception. The first aim will develop computational models of speech perception and test against behavioral data. Different models will be fit to speech perception data before, during and after exposure to the McGurk effect. Fitted models will be compared using held-out behavioral data. Because the models instantiate different theoretical constructs, model comparison will determine which explanatory constructs are essential. These results will provide a solid theoretical grounding for future studies, including those in Aims 2 and 3: searching for a neural correlate of an unjustified construct is likely to be fruitless. The second aim will examine speech perceptin through the lens of patterns of activity in STS/G measured with 7 tesla fMRI. We expect to observe reliable changes in STS/G response patterns before and after exposure to the McGurk effect, reflecting modification of speech representations (in contrast, in cortical areas driven solely by acoustic features, McGurk exposure should not change fMRI response patterns.) The third aim will use iEEG to record broadband high-frequency activity (BHA) from small populations of STS/G neurons with high temporal resolution. Responses to the auditory- only component of the McGurk speech (but not control speech) are predicted to show sustained decreases after successive blocks of audiovisual McGurk exposure, in lockstep with the perceptual development of FIR.