Multisensory Processing of Human Speech Measured with msec and mm Resolution

面对面的交流是人类最重要的互动形式。当我们交谈时，我们会收到 来自说话者声音的听觉信息和来自说话者面部的视觉信息。将这两者结合起来 信息的来源是困难的，因为它们到达得很快(大约每秒5个音节)和通信 说话者的声音和嘴巴动作之间是复杂的。我们建议研究 多感觉(听觉和视觉)言语感知的神经机制 脑电地形图(ECoG)是一种神经记录技术，将电极植入患者的大脑 癫痫患者。ECOG是我们研究问题的理想技术，因为它测量人脑 具有非常高的时间和空间分辨率(毫秒/毫米)的活动。ECOG可以用来检查 在语言感知过程中活跃的大脑区域的不同网络。听觉皮质中的电极强烈 由语音的听觉成分激活，而枕叶的电极被强烈激活 语音的视觉成分。在听觉和视觉皮质之间，后外侧颞叶皮质被认为是 将听觉和视觉语音结合起来。 听力不良是退伍军人最常见的残疾之一。因为言语是我们社会生活的基础 在人际关系中，言语感知能力差会导致社交孤立、抑郁和其他健康问题。一个 更好地理解多感官言语感知背后的神经机制将使我们能够 提高退伍军人的语言理解能力，使他们的生活质量得到重大改善。至 确保我们的结果立即适用于现实世界的情况，我们将研究大脑对 说英语的人说的自然的英语单词。 除了潜在的临床益处外，拟议的研究还将对基础 科学。多感觉整合是神经科学中的一个重要的新领域，已被证明是 大脑和行为的数学模型。我们的工作将成为实验和模型之间的桥梁 简单的多感官行为，如听觉-视觉定位和更复杂的认知任务， 以多感官言语感知为例。这些研究的成功完成将代表着 在我们对多感官语言感知的神经基础的理解上向前迈进了一大步。

Face-to-face communication is the most important form of human interaction. When conversing we receive auditory information from the talker's voice and visual information from the talker's face. Combining these two sources of information is difficult, as they arrive quickly (about 5 syllables per second) and the correspondence between the vocal sounds and the mouth movements made by the talker is complex. We propose to study the neural mechanisms that underlie multisensory (auditory and visual) speech perception using electrocorticography (ECoG), a neural recording technique in which electrodes are implanted in the brain of epileptic patients. ECoG is the ideal technique for our research question because it measures human brain activity with very high temporal and spatial resolution (millisecond/millimeter). ECoG can be used to examine the diverse network of brain areas active during speech perception. Electrodes in auditory cortex are strongly activated by the auditory component of speech, while electrodes in occipital lobe are strongly activated by the visual component of speech. Between auditory and visual cortex, posterior lateral temporal cortex is thought to integrate auditory and visual speech. Poor hearing is one of the most common disabilities in veterans. Since speech is the basis of our social relationships, poor speech perception can lead to social isolation, depression and other health problems. A better understanding of the neural mechanisms underlying multisensory speech perception will allow us to improve veterans' ability to understand speech, leading to major improvements in their quality of life. To ensure that our results are immediately applicable to real world situations, we will study brain responses to natural English language words spoken by English talkers. In addition to its potential clinical benefits, the proposed research will also have a significant impact on basic science. Multisensory integration is a major new field in neuroscience and has proven to be fertile ground for mathematical models of brain and behavior. Our work will serve as a bridge between experiments and models of simple multisensory behavior, such as auditory-visual localization and more complex cognitive tasks, exemplified by multisensory speech perception. The successful completion of these studies will represent a major step forward in our understanding of the neural substrates of multisensory speech perception.