Characterizing the recovery of spectral, temporal, and phonemic speech information from visual cues

表征从视觉线索中恢复频谱、时间和音位语音信息

• 批准号: 10563860
• 负责人: David Brang
• 金额: $ 55.04万
• 依托单位: UNIVERSITY OF MICHIGAN AT ANN ARBOR
• 依托单位国家: 美国
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-02-14 至 2028-01-31
• 项目状态: 未结题
• 来源: https://reporter.nih.gov/project-details/10563860
• 关键词: Acoustics; Attention; Auditory; Auditory area; Auditory system; Biological; Brain; Brain Injuries; Brain Neoplasms; Classification; Cochlear Implants; Code; Compensation; Crowding; Cues; Data; Development; Devices; Dimensions; Distributional Activity; Electrodes; Electroencephalography; Emotional; Frequencies; Functional Magnetic Resonance Imaging; Health; Hearing; Human; Illusions; Impairment; Individual; Lipreading; Maps; Measures; Modality; Modeling; Movement; Neurons; Noise; Oral; Oral cavity; Participant; Patients; Pattern; Perception; Periodicity; Physiological Processes; Population; Presbycusis; Process; Reaction Time; Recovery; Rehabilitation therapy; Research; Resolution; Resources; Route; Shapes; Signal Transduction; Social Interaction; Speech; Speech Perception; Speech Sound; Stimulus; Stroke; Superior temporal gyrus; System; Testing; Titrations; Training Programs; Trauma; Vision; Visual; Vocation; audiovisual speech; auditory stimulus; density; healthy aging; improved; neural prosthesis; neuromechanism; programs; response; restoration; sensory substitution; social; speech accuracy; visual information; visual speech

Project Summary Auditory speech perception is essential for social, vocational, and emotional health in hearing individuals. However, the reliability of auditory signals varies widely in everyday settings (e.g., at a crowded party), requiring supplemental processes to enable accurate speech perception. The principle mechanisms that support the perception of degraded auditory speech signals are auditory-visual (crossmodal) interactions, which can perceptually restore speech content using visual cues provided by lipreading, rhythmic articulatory movements, and the natural correlations present between oral resonance and mouth shape. Moreover, receptive speech processes can be limited through a variety of causes, including intrinsic brain tumor, stroke, cochlear implant usage, and age-related hearing loss, making compensatory crossmodal mechanisms necessary for one to continue working and maintaining healthy social interactions. However, the physiological processes that enable vision to facilitate speech perception remain poorly understood and no integrative model exists for how these multiple visual dimensions combine to enhance auditory speech perception. In the auditory domain, distributed populations of neurons encode spectro-temporal information about acoustic cues that are then transcoded into phonemes. We propose a dual-route perceptual model through which visual signals integrate with phoneme- coded neurons. First, a direct path through which viseme-to-phoneme conversions generate semi-overlapping distributions of activity in the superior temporal gyrus, leading to improved hearing through improved auditory phoneme tuning functions. Second, an indirect path through which visual features restore spectral information about speech frequencies and alter phoneme-response timing, resulting in improved auditory spectro-temporal profiles (which in turn are transcoded into phonemes with greater precision). Finally, we will examine the hypothesis that our perceptual system optimizes which of these visual dimensions is prioritized for recovery based on what is missing from the auditory signal. These studies will provide a unified framework for how speech perception benefits from different visual signals. By understanding biological approaches to crossmodally restoring degraded auditory speech information, we can develop better targeted rehabilitation programs and neural prostheses to maximize speech perception recovery after trauma or during healthy aging.

项目摘要 听觉言语知觉对于听力个体的社会、职业和情绪健康是必不可少的。 然而，听觉信号的可靠性在日常环境中(例如，在拥挤的派对上)变化很大，需要 补充流程，以实现准确的语音感知。支持该计划的主要机制 对退化的听觉语音信号的感知是听觉-视觉(交叉模式)交互作用，这可以 使用唇读提供的视觉提示、有节奏的发音动作 口腔共鸣与口型之间存在着天然的相关性。此外，接受性言语 过程可以通过各种原因来限制，包括固有的脑瘤、中风、人工耳蜗术 使用，以及与年龄相关的听力损失，使得代偿性跨模式机制对于一个人来说是必要的 继续工作并保持健康的社会交往。然而，能够使 视觉促进语音感知仍然知之甚少，而且还没有一个完整的模型来解释这些 多个视觉维度结合在一起，增强了听觉语音感知。在听觉领域，分布在 神经元群体编码关于声音提示的频谱-时间信息，然后将其代码转换为 音素。我们提出了一种双路径感知模型，通过该模型将视觉信号与音素整合在一起。 编码神经元。第一，视位到音位转换产生半重叠的直接途径 活动在颞上回的分布，通过改善听力而导致听力改善 音素调谐功能。第二，视觉特征恢复光谱信息的间接路径 关于语音频率和改变音素反应计时，从而改善听觉频谱时间 简档(转码为音素的精确度更高)。最后，我们将研究 假设我们的感知系统优化了这些视觉维度中的哪些是优先恢复的 基于从听觉信号中遗漏了什么。这些研究将为如何使用语音提供一个统一的框架 感知得益于不同的视觉信号。通过了解交叉模式的生物学方法 恢复退化的听觉语言信息，我们可以开发更有针对性的康复计划和 神经假体，以最大限度地提高创伤后或健康老龄化期间的语音感知恢复。