Multisensory Processing of Human Speech Measured with msec and mm Resolution

面对面的交流是人类互动的最重要形式。当我们交谈时， 来自说话者声音的听觉信息和来自说话者面部的视觉信息。结合这两 信息来源是困难的，因为它们到达得很快（大约每秒5个音节）， 说话者的声音和嘴部动作之间的关系是复杂的。我们建议研究 多感官（听觉和视觉）语言感知的神经机制， 皮质电图（ECoG），一种神经记录技术，其中电极植入大脑中， 癫痫患者ECoG是我们研究问题的理想技术，因为它可以测量人脑 活动具有非常高的时间和空间分辨率（毫秒/毫米）。ECoG可以用来检查 在言语感知过程中活跃的大脑区域的多样性网络。听觉皮层中的电极 由言语的听觉成分激活，而枕叶中的电极则由言语的听觉成分强烈激活 言语的视觉成分。在听觉和视觉皮层之间，后外侧颞叶皮层被认为是 整合听觉和视觉语音。 听力差是退伍军人中最常见的残疾之一。既然言语是我们社会生活的基础 在人际关系中，言语感知能力差会导致社交孤立、抑郁和其他健康问题。一 更好地理解多感官言语感知的神经机制将使我们能够 提高退伍军人理解语言的能力，从而大大改善他们的生活质量。到 为了确保我们的结果能立即应用于真实的世界情况，我们将研究大脑对 自然的英语单词说英语的人。 除了潜在的临床益处外，拟议的研究还将对基础研究产生重大影响。 科学多感觉整合是神经科学的一个重要新领域，已被证明是 大脑和行为的数学模型我们的工作将作为实验和模型之间的桥梁 简单的多感官行为，如视觉定位和更复杂的认知任务， 以多感官言语感知为例。这些研究的成功完成将代表一个 这是我们理解多感官言语感知的神经基础的一个重要步骤。

Face-to-face communication is the most important form of human interaction. When conversing we receive auditory information from the talker's voice and visual information from the talker's face. Combining these two sources of information is difficult, as they arrive quickly (about 5 syllables per second) and the correspondence between the vocal sounds and the mouth movements made by the talker is complex. We propose to study the neural mechanisms that underlie multisensory (auditory and visual) speech perception using electrocorticography (ECoG), a neural recording technique in which electrodes are implanted in the brain of epileptic patients. ECoG is the ideal technique for our research question because it measures human brain activity with very high temporal and spatial resolution (millisecond/millimeter). ECoG can be used to examine the diverse network of brain areas active during speech perception. Electrodes in auditory cortex are strongly activated by the auditory component of speech, while electrodes in occipital lobe are strongly activated by the visual component of speech. Between auditory and visual cortex, posterior lateral temporal cortex is thought to integrate auditory and visual speech. Poor hearing is one of the most common disabilities in veterans. Since speech is the basis of our social relationships, poor speech perception can lead to social isolation, depression and other health problems. A better understanding of the neural mechanisms underlying multisensory speech perception will allow us to improve veterans' ability to understand speech, leading to major improvements in their quality of life. To ensure that our results are immediately applicable to real world situations, we will study brain responses to natural English language words spoken by English talkers. In addition to its potential clinical benefits, the proposed research will also have a significant impact on basic science. Multisensory integration is a major new field in neuroscience and has proven to be fertile ground for mathematical models of brain and behavior. Our work will serve as a bridge between experiments and models of simple multisensory behavior, such as auditory-visual localization and more complex cognitive tasks, exemplified by multisensory speech perception. The successful completion of these studies will represent a major step forward in our understanding of the neural substrates of multisensory speech perception.