Natural audiovisual speech encoding in the early stages of the human cortical hierarchy

人类皮质层次结构早期阶段的自然视听语音编码

• 批准号: 10357771
• 负责人: Edmund Lalor
• 金额: $ 38.5万
• 依托单位: UNIVERSITY OF ROCHESTER
• 依托单位国家: 美国
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-03-15 至 2024-02-29
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/10357771
• 关键词: Acoustics; Affect; Area; Articulation; Attention; Auditory; Auditory area; Brain; Categories; Clinical; Clinical Research; Complex; Comprehension; Cueing for speech; Data; Disease; Electrocorticogram; Electroencephalography; Environment; Event-Related Potentials; Eyebrow structure; Face; Functional Magnetic Resonance Imaging; Future; Goals; Head Movements; Human; Impairment; Knowledge; Learning; Life; Linguistics; Lip structure; Lipreading; Measures; Mental disorders; Methods; Modeling; Motion; Neurobiology; Noise; Pathway interactions; Perception; Population; Process; Property; Publishing; Reporting; Research; Sampling; Schizophrenia; Signal Transduction; Speech; Speech Perception; Stimulus; Stream; Study models; Techniques; Testing; Vision; Visual; Visual Cortex; Work; audiovisual speech; autism spectrum disorder; base; cohort; developmental disease; experience; experimental study; flexibility; hemodynamics; improved; indexing; insight; language processing; multisensory; neuromechanism; neurophysiology; nonhuman primate; novel; novel strategies; patient population; relating to nervous system; sound; speech processing; temporal measurement; tool; visual information; visual speech

PROJECT SUMMARY Speech is central to human life. Yet how the human brain processes speech in complex everyday situations remains poorly understood. One prominent idea is that speech perception is carried out using brain areas and mechanisms that are used for processing sounds more generally. And it has been suggested that these mechanisms become specialized for speech through learning, resulting in a speech processing network in the brain that processes increasingly complex aspects of the speech signal at successive hierarchical stages. But questions about the function of this hierarchy remain. In particular, while it is commonly acknowledged that seeing a speaker’s face in noisy environments can improve comprehension, our understanding of how visual speech influences the hierarchical processing of speech remain unclear. This is unfortunate as speech processing, and multisensory speech processing in particular, have been reported to be affected in a number of clinical disorders, including autism and schizophrenia. Thus, as well as contributing to our understanding of this most fundamental of human abilities, better knowledge of the neural mechanisms underpinning audiovisual speech processing could have important clinical research implications. One of the principal reasons for our lack of knowledge on the neurophysiology of audiovisual speech is the technical challenge associated with indexing the neural processing of natural speech with high temporal resolution and at multiple levels of the speech processing hierarchy. Non-human primates represent a less than perfect model for studying human speech processing, the hemodynamic changes underlying functional magnetic resonance imaging are too slow to track natural speech dynamics, and electrocorticography samples only a limited number of brain areas and cannot be broadly applied in clinical research. Recently, our group has introduced several new approaches for indexing natural speech processing using electroencephalography (EEG). These include entirely novel frameworks for producing dependent measures of the hierachical encoding of natural speech, and for quantifying multisensory integration of natural audiovisual speech. The present proposal seeks to exploit this opportunity to test the hypothesis that the integration of audio and visual speech is a flexible, multistage process that adapts to optimize comprehension based on the current listening conditions. Across three objectives the proposal aims to characterize this flexibility by determining how the hierarchical processing stage at which visual and audio speech are integrated varies as a function of 1) the listening environment, 2) the visual information available and 3) the deployment of attention. The work promises to bring a new depth of understanding to the perception of one of humanity’s most essential signals. And it will introduce several novel analyses and experimental paradigms that should be easily deployable in tackling research on clinical cohorts in which speech processing and/or multisensory integration is impaired.

项目总结 语言是人类生活的中心。然而，人类大脑是如何在复杂的日常情况下处理语音的 人们对此仍然知之甚少。一个突出的观点是，言语感知是使用大脑区域和 更一般地用于处理声音的机制。有人提出，这些 通过学习，语音处理机制变得专门化，从而在 在连续的层次结构阶段处理日益复杂的语音信号的大脑。但 关于这个等级制度的功能的问题仍然存在。特别是，尽管人们普遍认为 在嘈杂的环境中看到说话者的脸可以提高我们对视觉效果的理解 语音对语音的分层加工的影响目前还不清楚。作为演讲，这是很不幸的 据报道，处理，特别是多感官语言处理，在许多情况下会受到影响 临床障碍，包括自闭症和精神分裂症。因此，不仅有助于我们对这一点的理解 人类最基本的能力，更好地了解支撑视听的神经机制 语音处理可能具有重要的临床研究意义。我们缺乏的主要原因之一是 视听语音的神经生理学知识是与索引相关的技术挑战 自然语音的多层次高时间分辨率神经处理 处理层次结构。非人灵长类动物代表了一种不太完美的研究人类语言的模型 在处理过程中，功能磁共振成像背后的血流动力学变化太慢，无法跟踪 自然语音动力学和皮质脑电图术仅对有限数量的大脑区域进行采样，不能 广泛应用于临床研究。最近，我们小组引入了几种新的索引方法 使用脑电(EEG)进行自然语音处理。其中包括全新的框架，用于 产生自然语音的分级编码的依赖测量，并用于量化多感官 自然视听语音的融合。目前的建议旨在利用这个机会来测试 假设视听语音的整合是一个灵活的、多阶段的过程，适合于优化 基于当前听力条件的理解。该提案旨在实现三个目标 通过确定视觉和音频所处的层次处理阶段如何表征这种灵活性 作为1)收听环境、2)可用的可视信息以及 3)注意的部署。这部作品有望给人们带来新的理解 人类最基本的信号之一。并将介绍几种新颖的分析和实验 在处理语音处理的临床队列研究中应易于部署的范例 和/或多感官整合受损。

项目成果

The effect of gaze on EEG measures of multisensory integration in a cocktail party scenario.

鸡尾酒会场景中凝视对多感官整合脑电图测量的影响。

• DOI: 10.1101/2023.08.23.554451
• 发表时间: 2023
• 期刊: bioRxiv : the preprint server for biology
• 作者: Ahmed,Farhin;Nidiffer,AaronR;Lalor,EdmundC
• 通讯作者: Lalor,EdmundC

Attention Differentially Affects Acoustic and Phonetic Feature Encoding in a Multispeaker Environment.

• DOI: 10.1523/jneurosci.1455-20.2021
• 发表时间: 2022-01-26
• 期刊: The Journal of neuroscience : the official journal of the Society for Neuroscience
• 作者: Teoh ES;Ahmed F;Lalor EC
• 通讯作者: Lalor EC