NEURAL SUBSTRATES OF OPTIMAL MULTISENSORY INTEGRATION

 说明书(申请人提供)言语感知是人脑进行的最重要的认知操作之一，从根本上讲是多感官的：当我们与某人交谈时，我们既使用他们面部的视觉信息，也使用他们的声音的听觉信息。当语音的听觉成分有噪声时，无论是由于听力障碍还是正常衰老，多感官言语感知尤其重要。然而，与听觉言语知觉相比，人们对视觉言语感知背后的神经计算的了解要少得多。为了弥补现有知识中的这一差距，我们将使用来自两种互补的大脑活动测量方法--BOLD功能磁共振成像和皮层脑电成像(ECoG)的融合证据。这些神经记录研究的结果将在一个灵活的计算模型的背景下进行解释，该模型基于一个新兴的原则，即大脑使用最佳或贝叶斯推理来执行多感觉整合，将当前可用的感觉信息与先前的经验相结合。在第一个目标中，将构建一个贝叶斯模型来解释多感官言语知觉的个体差异，沿着三个轴：受试者理解嘈杂视听言语的能力；受试者对麦格克效应的敏感性，一种多感官错觉；以及花在凝视上的时间 会说话的脸的嘴巴。在第二个目标中，我们将使用BOLD功能磁共振信号的体素前向编码模型来探索视觉语音的神经编码。我们将开发编码模型来测试语言学和计算机视觉文献中的7种不同的视觉语音表征理论。在第三个目标中，我们将在目标1中开发的贝叶斯模型的指导下，使用ECoG来检查整合视觉和听觉语音的神经计算。首先，我们将研究由我们的模型预测的多感觉语音的神经变异性降低。其次，我们将研究单感和多感言语的表征空间。

 DESCRIPTION (provided by applicant) Speech perception is one of the most important cognitive operations performed by the human brain and is fundamentally multisensory: when conversing with someone, we use both visual information from their face and auditory information from their voice. Multisensory speech perception is especially important when the auditory component of the speech is noisy, either due to a hearing disorder or normal aging. However, much less is known about the neural computations underlying visual speech perception than about those underlying auditory speech perception. To remedy this gap in existing knowledge, we will use converging evidence from two complementary measures of brain activity, BOLD fMRI and electrocorticography (ECoG). The results of these neural recording studies will be interpreted in the context of a flexible computational model based on the emerging tenet that the brain performs multisensory integration using optimal or Bayesian inference, combining the currently available sensory information with prior experience. In the first Aim, a Bayesian model will be constructed to explain individual differences in multisensory speech perception along three axes: subjects' ability to understand noisy audiovisual speech; subjects' susceptibility to the McGurk effect, a multisensory illusion; and the time spent fixating the mouth of a talking face. In the second Aim, we will explore the neural encoding of visual speech using voxel-wise forward encoding models of the BOLD fMRI signal. We will develop encoding models to test 7 different theories of visual speech representation from the linguistic and computer vision literature. In the third Aim, we will use ECoG to examine the neural computations for integrating visual and auditory speech, guided by the Bayesian models developed in Aim 1. First, we will study reduced neural variability for multisensory speech predicted by our model. Second, we will study the representational space of unisensory and multisensory speech.