Audiovisual Speech Perception

点击翻译按钮获取中文摘要

Understanding speech is a vital part of our daily life, for social, emotional, and informational purposes. Although audition plays an important role in perceiving speech information, vision is also an integral part of this process. It has been recognized for many years that viewing someone’s face during a conversation can boost our speech comprehension. Thus, speech perception, in its most natural form (face-to-face), is visual as well as auditory. This grant will support a research program that aims at understanding how the visual and acoustic linguistic aspects of speech are bound during the perception of spoken language. Specifically, during this grant period, my students and I will explore the perceptual and neural mechanisms involved in audiovisual speech perception using three methodological approaches. The first involves studies that address the temporal window over which audiovisual speech integration occurs. Since the invention of talking movies, researchers and filmmakers have known that strict temporal overlapping between the video and sound is by no means necessary for people to perceive them as a single event. In other words, people cannot really tell the tracks are out-of-sync over a broad range of temporal offsets. We will explore this phenomenon as a method to understand how the nervous system combines information from different senses. The second component addresses the role of gaze fixations on audiovisual speech perception. Active vision determines what parts of the face are selectively processed and these studies address how this selection influences speech perception. Specifically, we want to determine how information gathering (gaze behaviour) is influenced by the temporal synchrony between the modalities. The final research area involves the use of functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) to study the neural representation of audiovisual speech. The speech stimuli used in all components of the research program are created using innovative animation and computer-based editing procedures. The overall research program will make significant contributions to our understanding of speech information processing and the neurological mechanisms that carry out this remarkable perceptual skill. In recent years, we have used our controlled animated stimuli to determine the temporal and spatial conditions required for audiovisual integration. With fMRI and the study of neurological patients we have begun to identify the complex network of neurological structures that contribute to this perception. Because the program combines different methodologies and techniques, it provides a rich training environment for students.