Audiovisual Speech Perception

理解言语是我们日常生活的重要组成部分，用于社交，情感和信息目的。虽然听觉在感知语音信息中起着重要的作用，但视觉也是这一过程中不可或缺的一部分。多年来，人们已经认识到，在谈话中观察某人的脸可以提高我们的言语理解力。因此，言语感知，在其最自然的形式（面对面），是视觉和听觉。这笔赠款将支持一项研究计划，旨在了解语音的视觉和听觉语言方面是如何在口语的感知过程中结合起来的。具体来说，在此期间，我的学生和我将探讨知觉和神经机制参与视听言语知觉使用三种方法。第一个涉及解决视听语音整合发生的时间窗口的研究。自从有声电影发明以来，研究人员和电影制作人就知道，视频和声音之间严格的时间重叠对于人们将它们视为单一事件来说是不必要的。换句话说，人们不能真正告诉轨道是在很大范围的时间偏移不同步。我们将探索这种现象，作为理解神经系统如何结合来自不同感官的信息的方法。第二个组成部分解决视听语音感知的注视注视的作用。主动视觉决定了面部的哪些部分被选择性地处理，这些研究解决了这种选择如何影响言语感知。具体来说，我们要确定如何信息收集（凝视行为）的影响，由模态之间的时间同步。最后一个研究领域涉及使用功能性磁共振成像（fMRI）和脑电图（EEG）来研究视听语音的神经表征。在研究计划的所有组成部分中使用的语音刺激是使用创新的动画和基于计算机的编辑程序创建的。整个研究计划将为我们理解语音信息处理和执行这种显着感知技能的神经机制做出重大贡献。近年来，我们已经使用我们控制的动画刺激，以确定视听整合所需的时间和空间条件。通过功能性磁共振成像和对神经系统患者的研究，我们已经开始识别有助于这种感知的复杂神经结构网络。由于该计划结合了不同的方法和技术，它为学生提供了丰富的培训环境。

Understanding speech is a vital part of our daily life, for social, emotional, and informational purposes. Although audition plays an important role in perceiving speech information, vision is also an integral part of this process. It has been recognized for many years that viewing someone’s face during a conversation can boost our speech comprehension. Thus, speech perception, in its most natural form (face-to-face), is visual as well as auditory. This grant will support a research program that aims at understanding how the visual and acoustic linguistic aspects of speech are bound during the perception of spoken language. Specifically, during this grant period, my students and I will explore the perceptual and neural mechanisms involved in audiovisual speech perception using three methodological approaches. The first involves studies that address the temporal window over which audiovisual speech integration occurs. Since the invention of talking movies, researchers and filmmakers have known that strict temporal overlapping between the video and sound is by no means necessary for people to perceive them as a single event. In other words, people cannot really tell the tracks are out-of-sync over a broad range of temporal offsets. We will explore this phenomenon as a method to understand how the nervous system combines information from different senses. The second component addresses the role of gaze fixations on audiovisual speech perception. Active vision determines what parts of the face are selectively processed and these studies address how this selection influences speech perception. Specifically, we want to determine how information gathering (gaze behaviour) is influenced by the temporal synchrony between the modalities. The final research area involves the use of functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) to study the neural representation of audiovisual speech. The speech stimuli used in all components of the research program are created using innovative animation and computer-based editing procedures. The overall research program will make significant contributions to our understanding of speech information processing and the neurological mechanisms that carry out this remarkable perceptual skill. In recent years, we have used our controlled animated stimuli to determine the temporal and spatial conditions required for audiovisual integration. With fMRI and the study of neurological patients we have begun to identify the complex network of neurological structures that contribute to this perception. Because the program combines different methodologies and techniques, it provides a rich training environment for students.