Integration and Enhancement in Audiovisual Speech Perception

视听语音感知的整合和增强

• 批准号: 0214224
• 负责人: Lynne Bernstein
• 金额: $ 34.61万
• 依托单位: House Ear Institute
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2002
• 资助国家: 美国
• 起止时间: 2002-08-01 至 2008-01-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0214224&HistoricalAwards=false
• 关键词: Integration; Enhancement; Audiovisual; Speech; Perception

With National Science Foundation support, Dr. Lynne E. Bernstein will conduct two years of research into the underlying brain mechanisms responsible for the human ability to combine speech information that is heard and speech information that is observed by watching a talker's face. The focus will be to explain the fact that being able to see a talker under noisy conditions dramatically improves the ability to hear that talker's speech. When measured, this effect is equivalent to almost quadrupling the loudness of the speech signal. A fundamental question is whether this effect occurs because listeners correlate speech information from the talker's lips and face with speech sounds, or whether the effect occurs whenever a visual object is paired with speech. Two main experiments will be done. The first will measure speech detectability levels in noise. It will compare perception of speech in noise with the same speech in noise paired with three types of stimuli: (1) a talking face, (2) a static but temporally aligned ellipse, (3) a dynamic ellipse whose vertical extent is controlled by the loudness of the speech signal. If the mere overlap between heard speech and a visual object results in improved perception, then the brain appears to turn up the gain when events overlap, regardless of whether they have similar significance. If the static ellipse is not effective but the dynamic one is, then the brain appears to depend on a correlation between stimuli but one that does not require the visual stimuli to be speech. But if only the talking face is effective, it is likely that the brain solves the noise problem by using processes that are specialized for speech. In the second experiment, electrophysiological recordings of brain activity will be made during a task like that in the first experiment. This experiment asks whether seeing a talker is the same as turning up the gain in the sound, as far as the brain is concerned. The event-related potentials to be obtained will be analyzed to find when brain events occurred and where in the brain they occurred. Other analyses will be used to investigate how the visual and auditory processing areas of the brain synchronize their activity during the detection of speech sounds.How the brain combines information from different sensory-perceptual modalities is one of the great mysteries of human perception, along with whether speech is processed by the brain in the same or a different manner than it processes other types of stimuli. This project is among the first to use both brain and behavioral methods to investigate how the brain combines auditory and visual speech under noisy conditions. Until recently, the techniques to study two senses at once were not available, and so little research investigated how the brain creates a coherent perception of the world from the diverse information it gets. Knowledge to be obtained will have practical implications. For example, it can suggest how visual stimuli can help listeners to get critical information under noisy conditions such as an airplane cockpit. It can help explain why people with hearing impairments benefit from being able to communicate face-to-face. The knowledge can also be extended to developmental research to determine why children have more difficulty than adults when listening to speech under noisy conditions, such as a noisy classroom.

在国家科学基金会的支持下，林恩·E·伯恩斯坦博士将进行为期两年的研究，研究人类将听到的语音信息和通过观察说话者的面部观察到的语音信息结合起来的潜在大脑机制。重点将解释这样一个事实，即在嘈杂的条件下能够看到说话者，大大提高了听到说话者的演讲的能力。当测量时，这种效果相当于几乎将语音信号的响度提高了四倍。一个根本的问题是，这种效果的产生是因为听者将说话者嘴唇和面部的语音信息与语音相关联，还是当视觉对象与语音配对时，这种效果就会发生。将进行两个主要的实验。第一项测试将测量噪声中的语音可检测性水平。它将比较在噪声中的语音感知和在噪声中的相同语音与三种类型的刺激配对的噪声中的语音：(1)说话的脸，(2)静态的但在时间上对齐的椭圆，(3)其垂直程度由语音信号的响度控制的动态椭圆。如果仅仅是听到的语音和视觉对象之间的重叠就能改善知觉，那么当事件重叠时，大脑似乎会增加收益，而不管它们是否具有类似的意义。如果静态椭圆无效，而动态椭圆有效，那么大脑似乎依赖于刺激之间的相关性，但并不要求视觉刺激是语音。但是，如果只有说话的脸是有效的，那么大脑很可能通过使用专门用于说话的过程来解决噪音问题。在第二个实验中，将在与第一个实验类似的任务中对大脑活动进行电生理记录。这项实验的问题是，就大脑而言，看到说话者是否等同于提高声音的增益。将对获得的事件相关电位进行分析，以找出大脑事件发生的时间和它们在大脑中的位置。其他分析将被用来研究大脑的视觉和听觉处理区域如何在检测语音的过程中同步它们的活动。大脑如何结合来自不同感觉-感知模式的信息是人类感知的最大谜团之一，以及大脑处理语言的方式与处理其他类型刺激的方式是相同还是不同。这个项目是第一个同时使用大脑和行为方法来研究大脑在嘈杂条件下如何结合听觉和视觉语言的项目之一。直到最近，同时研究两种感官的技术还不存在，所以很少有研究研究大脑如何从它获得的各种信息中创造出对世界的连贯感知。将获得的知识将具有实际意义。例如，它可以建议视觉刺激如何帮助听众在飞机驾驶舱等嘈杂条件下获得关键信息。这有助于解释为什么听力障碍的人能从面对面的交流中受益。这一知识还可以扩展到发展研究，以确定为什么儿童在嘈杂的条件下(如嘈杂的教室)听演讲时比成年人更困难。