Integration and Enhancement in Audiovisual Speech Perception

在美国国家科学基金会（National Science Foundation）的支持下，林恩·伯恩斯坦（Lynne E. Bernstein）博士将进行为期两年的研究，研究人类将听到的语音信息与通过观察说话人的面部观察到的语音信息结合起来的能力的潜在大脑机制。重点将是解释这样一个事实，即能够在嘈杂的条件下看到说话人，可以显著提高听到说话人讲话的能力。当测量时，这种效果相当于语音信号的响度几乎增加了四倍。一个基本的问题是，这种效应的产生是因为听者将说话者的嘴唇和面部的语音信息与语音联系起来，还是因为视觉对象与语音配对时，这种效应就会发生。将进行两个主要实验。第一个将测量噪声中的语音可检测水平。它将比较噪声中的语音感知与在噪声中与三种类型的刺激配对的相同语音感知：(1)说话的脸，(2)静态但暂时对齐的椭圆，(3)垂直程度由语音信号的响度控制的动态椭圆。如果仅仅是听到的话语和看到的物体之间的重叠就能提高感知能力，那么当事件重叠时，不管它们是否具有相似的意义，大脑似乎都会提高感知能力。如果静态椭圆无效，而动态椭圆有效，那么大脑似乎依赖于刺激之间的相关性，但这种相关性并不要求视觉刺激是语音。但是，如果只有说话的脸是有效的，那么很可能大脑通过使用专门用于言语的过程来解决噪音问题。在第二个实验中，在完成与第一个实验类似的任务时，将对大脑活动进行电生理记录。这个实验问的是，就大脑而言，看到一个说话的人是否等同于打开声音的增益。将分析获得的事件相关电位，以找出大脑事件发生的时间和位置。其他分析将用于研究大脑的视觉和听觉处理区域如何在语音检测过程中同步其活动。大脑如何结合来自不同感官知觉模式的信息，是人类感知的一大谜团之一，此外，大脑处理语音的方式与处理其他类型刺激的方式是相同还是不同。这个项目是第一个使用大脑和行为方法来研究大脑如何在嘈杂条件下结合听觉和视觉语言的项目之一。直到最近，同时研究两种感官的技术还不存在，因此很少有研究调查大脑是如何从它获得的各种信息中创造出对世界的连贯感知的。所获得的知识将具有实际意义。例如，它可以表明视觉刺激如何帮助听众在嘈杂的条件下（如飞机驾驶舱）获得关键信息。这有助于解释为什么有听力障碍的人能从面对面的交流中受益。这些知识也可以扩展到发展研究，以确定为什么儿童在嘈杂的条件下（如嘈杂的教室）听讲话比成人更困难。

With National Science Foundation support, Dr. Lynne E. Bernstein will conduct two years of research into the underlying brain mechanisms responsible for the human ability to combine speech information that is heard and speech information that is observed by watching a talker's face. The focus will be to explain the fact that being able to see a talker under noisy conditions dramatically improves the ability to hear that talker's speech. When measured, this effect is equivalent to almost quadrupling the loudness of the speech signal. A fundamental question is whether this effect occurs because listeners correlate speech information from the talker's lips and face with speech sounds, or whether the effect occurs whenever a visual object is paired with speech. Two main experiments will be done. The first will measure speech detectability levels in noise. It will compare perception of speech in noise with the same speech in noise paired with three types of stimuli: (1) a talking face, (2) a static but temporally aligned ellipse, (3) a dynamic ellipse whose vertical extent is controlled by the loudness of the speech signal. If the mere overlap between heard speech and a visual object results in improved perception, then the brain appears to turn up the gain when events overlap, regardless of whether they have similar significance. If the static ellipse is not effective but the dynamic one is, then the brain appears to depend on a correlation between stimuli but one that does not require the visual stimuli to be speech. But if only the talking face is effective, it is likely that the brain solves the noise problem by using processes that are specialized for speech. In the second experiment, electrophysiological recordings of brain activity will be made during a task like that in the first experiment. This experiment asks whether seeing a talker is the same as turning up the gain in the sound, as far as the brain is concerned. The event-related potentials to be obtained will be analyzed to find when brain events occurred and where in the brain they occurred. Other analyses will be used to investigate how the visual and auditory processing areas of the brain synchronize their activity during the detection of speech sounds.How the brain combines information from different sensory-perceptual modalities is one of the great mysteries of human perception, along with whether speech is processed by the brain in the same or a different manner than it processes other types of stimuli. This project is among the first to use both brain and behavioral methods to investigate how the brain combines auditory and visual speech under noisy conditions. Until recently, the techniques to study two senses at once were not available, and so little research investigated how the brain creates a coherent perception of the world from the diverse information it gets. Knowledge to be obtained will have practical implications. For example, it can suggest how visual stimuli can help listeners to get critical information under noisy conditions such as an airplane cockpit. It can help explain why people with hearing impairments benefit from being able to communicate face-to-face. The knowledge can also be extended to developmental research to determine why children have more difficulty than adults when listening to speech under noisy conditions, such as a noisy classroom.