NEURAL SUBSTRATES OF OPTIMAL MULTISENSORY INTEGRATION

最佳多感官整合的神经基质

• 批准号: 9197698
• 负责人: Michael S Beauchamp
• 金额: $ 34.67万
• 依托单位: BAYLOR COLLEGE OF MEDICINE
• 依托单位国家: 美国
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-01-01 至 2020-12-31
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/9197698
• 关键词: Auditory; Auditory Perception; Bayesian Analysis; Bayesian Modeling; Behavioral; Brain; Brain imaging; Brain region; Clinical; Cognitive; Computer Simulation; Computer Vision Systems; Data; Electrocorticogram; Employee Strikes; Eye; Eye Movements; Face; Functional Magnetic Resonance Imaging; Hearing problem; Human; Illusions; Individual; Individual Differences; Investigation; Knowledge; Language; Left; Linguistics; Link; Literature; Measures; Mediating; Modality; Modeling; Movement; Neurons; Noise; Oral cavity; Perception; Persons; Population; Predisposition; Property; Publishing; Sample Size; Sensory; Signal Transduction; Speech; Speech Perception; Stimulus; Structure of superior temporal sulcus; Techniques; Testing; Time; Visual; Vocabulary; Voice; audiovisual speech; base; behavior measurement; experience; flexibility; hearing impairment; multisensory; neural model; normal aging; operation; predictive modeling; public health relevance; relating to nervous system; response; sample fixation; speech accuracy; theories; visual information; visual speech

 DESCRIPTION (provided by applicant) Speech perception is one of the most important cognitive operations performed by the human brain and is fundamentally multisensory: when conversing with someone, we use both visual information from their face and auditory information from their voice. Multisensory speech perception is especially important when the auditory component of the speech is noisy, either due to a hearing disorder or normal aging. However, much less is known about the neural computations underlying visual speech perception than about those underlying auditory speech perception. To remedy this gap in existing knowledge, we will use converging evidence from two complementary measures of brain activity, BOLD fMRI and electrocorticography (ECoG). The results of these neural recording studies will be interpreted in the context of a flexible computational model based on the emerging tenet that the brain performs multisensory integration using optimal or Bayesian inference, combining the currently available sensory information with prior experience. In the first Aim, a Bayesian model will be constructed to explain individual differences in multisensory speech perception along three axes: subjects' ability to understand noisy audiovisual speech; subjects' susceptibility to the McGurk effect, a multisensory illusion; and the time spent fixating the mouth of a talking face. In the second Aim, we will explore the neural encoding of visual speech using voxel-wise forward encoding models of the BOLD fMRI signal. We will develop encoding models to test 7 different theories of visual speech representation from the linguistic and computer vision literature. In the third Aim, we will use ECoG to examine the neural computations for integrating visual and auditory speech, guided by the Bayesian models developed in Aim 1. First, we will study reduced neural variability for multisensory speech predicted by our model. Second, we will study the representational space of unisensory and multisensory speech.

 描述（由申请人提供）言语感知是由人类大脑执行的最重要的认知操作之一，并且基本上是多感官的：当与某人交谈时，我们使用来自他们的面部的视觉信息和来自他们的声音的听觉信息。当语音的听觉成分由于听力障碍或正常老化而嘈杂时，多感官语音感知尤其重要。然而，对视觉语音感知的神经计算知之甚少，而对听觉语音感知的神经计算知之甚少。为了弥补现有知识中的这一差距，我们将使用来自两种互补的大脑活动测量方法（BOLD fMRI和ECoG）的融合证据。这些神经记录研究的结果将在灵活的计算模型的背景下进行解释，该模型基于新兴的原则，即大脑使用最佳或贝叶斯推理进行多感官整合，将当前可用的感官信息与先前的经验相结合。 在第一个目标中，将构建一个贝叶斯模型来解释多感官言语感知的个体差异沿着三个轴是：受试者理解嘈杂视听言语的能力;受试者对麦格克效应（一种多感官错觉）的易感性;以及注视时间 一张会说话的脸。 在第二个目标中，我们将探索视觉语音的神经编码使用体素的前向编码模型的BOLD功能磁共振成像信号。我们将开发编码模型来测试来自语言学和计算机视觉文献的7种不同的视觉语音表示理论。在第三个目标中，我们将使用ECoG来检查在目标1中开发的贝叶斯模型的指导下整合视觉和听觉语音的神经计算。首先，我们将研究我们的模型预测的多感官语音的神经变异性降低。其次，我们将研究单感官和多感官言语的表征空间。