Natural audiovisual speech encoding in the early stages of the human cortical hierarchy

项目摘要 言语是人类生活的中心。然而，在复杂的日常情况下， 仍然知之甚少。一个突出的观点是，言语感知是使用大脑区域进行的， 更普遍地用于处理声音的机制。有人认为， 通过学习，机制变得专门用于语音，从而在语音处理网络中产生语音处理网络。 在连续的层次阶段处理语音信号的日益复杂的方面的大脑。但 关于这一等级制度的功能问题仍然存在。特别是，虽然人们普遍承认， 在嘈杂的环境中看到说话者的脸可以提高理解力， 言语影响言语的层次加工仍不清楚。不幸的是， 处理，特别是多感官语音处理，据报道在许多情况下都会受到影响。 临床疾病，包括自闭症和精神分裂症。因此，除了有助于我们理解这一点外， 人类最基本的能力，更好地了解支持视听的神经机制 语音处理可能具有重要的临床研究意义。我们缺乏的主要原因之一 视听语音的神经生理学知识是与索引相关的技术挑战 对自然语音的神经处理，具有高时间分辨率和语音的多个层次 处理层次结构。非人类灵长类动物是研究人类语言的一个不太完美的模型 处理，功能磁共振成像的血液动力学变化太慢，无法跟踪 自然语言动力学和皮质电描记术仅对有限数量的大脑区域进行采样， 广泛应用于临床研究。最近，我们的小组介绍了几种新的索引方法 使用脑电图（EEG）的自然语音处理。其中包括全新的框架， 产生自然语音的分级编码的相关测量，以及用于量化多感觉 自然视听语音的整合。本提案旨在利用这一机会测试 假设音频和视频语音的整合是一个灵活的，多阶段的过程，适应优化 根据当前的听力条件。该提案旨在实现三个目标， 通过确定视频和音频的分层处理阶段如何描述这种灵活性， 语音被整合作为1）收听环境，2）可用的视觉信息和 3)注意力的部署。这项工作有望带来一个新的理解深度的看法， 人类最重要的信号之一并将介绍几种新颖的分析和实验 范例，应该很容易部署在解决临床队列研究，其中语音处理 和/或多感觉整合受损。

PROJECT SUMMARY Speech is central to human life. Yet how the human brain processes speech in complex everyday situations remains poorly understood. One prominent idea is that speech perception is carried out using brain areas and mechanisms that are used for processing sounds more generally. And it has been suggested that these mechanisms become specialized for speech through learning, resulting in a speech processing network in the brain that processes increasingly complex aspects of the speech signal at successive hierarchical stages. But questions about the function of this hierarchy remain. In particular, while it is commonly acknowledged that seeing a speaker’s face in noisy environments can improve comprehension, our understanding of how visual speech influences the hierarchical processing of speech remain unclear. This is unfortunate as speech processing, and multisensory speech processing in particular, have been reported to be affected in a number of clinical disorders, including autism and schizophrenia. Thus, as well as contributing to our understanding of this most fundamental of human abilities, better knowledge of the neural mechanisms underpinning audiovisual speech processing could have important clinical research implications. One of the principal reasons for our lack of knowledge on the neurophysiology of audiovisual speech is the technical challenge associated with indexing the neural processing of natural speech with high temporal resolution and at multiple levels of the speech processing hierarchy. Non-human primates represent a less than perfect model for studying human speech processing, the hemodynamic changes underlying functional magnetic resonance imaging are too slow to track natural speech dynamics, and electrocorticography samples only a limited number of brain areas and cannot be broadly applied in clinical research. Recently, our group has introduced several new approaches for indexing natural speech processing using electroencephalography (EEG). These include entirely novel frameworks for producing dependent measures of the hierachical encoding of natural speech, and for quantifying multisensory integration of natural audiovisual speech. The present proposal seeks to exploit this opportunity to test the hypothesis that the integration of audio and visual speech is a flexible, multistage process that adapts to optimize comprehension based on the current listening conditions. Across three objectives the proposal aims to characterize this flexibility by determining how the hierarchical processing stage at which visual and audio speech are integrated varies as a function of 1) the listening environment, 2) the visual information available and 3) the deployment of attention. The work promises to bring a new depth of understanding to the perception of one of humanity’s most essential signals. And it will introduce several novel analyses and experimental paradigms that should be easily deployable in tackling research on clinical cohorts in which speech processing and/or multisensory integration is impaired.