Audiovisual Speech Perception

视听语音感知

• 批准号: RGPIN-2014-04985
• 负责人: Munhall, Kevin
• 金额: $ 2.48万
• 依托单位: Queen's University
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2018
• 资助国家: 加拿大
• 起止时间: 2018-01-01 至 2019-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=660430
• 关键词: Audiovisual; Speech; Perception

Understanding speech is a vital part of our daily life, for social, emotional, and informational purposes. Although audition plays an important role in perceiving speech information, vision is also an integral part of this process. It has been recognized for many years that viewing someone's face during a conversation can boost our speech comprehension. Thus, speech perception, in its most natural form (face-to-face), is visual as well as auditory. This grant will support a research program that aims at understanding how the visual and acoustic linguistic aspects of speech are bound during the perception of spoken language. **Specifically, during this grant period, my students and I will explore the perceptual and neural mechanisms involved in audiovisual speech perception using three methodological approaches. The first involves studies that address the temporal window over which audiovisual speech integration occurs. Since the invention of talking movies, researchers and filmmakers have known that strict temporal overlapping between the video and sound is by no means necessary for people to perceive them as a single event. In other words, people cannot really tell the tracks are out-of-sync over a broad range of temporal offsets. We will explore this phenomenon as a method to understand how the nervous system combines information from different senses. The second component addresses the role of gaze fixations on audiovisual speech perception. Active vision determines what parts of the face are selectively processed and these studies address how this selection influences speech perception. Specifically, we want to determine how information gathering (gaze behaviour) is influenced by the temporal synchrony between the modalities. The final research area involves the use of functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) to study the neural representation of audiovisual speech. The speech stimuli used in all components of the research program are created using innovative animation and computer-based editing procedures. **The overall research program will make significant contributions to our understanding of speech information processing and the neurological mechanisms that carry out this remarkable perceptual skill. In recent years, we have used our controlled animated stimuli to determine the temporal and spatial conditions required for audiovisual integration. With fMRI and the study of neurological patients we have begun to identify the complex network of neurological structures that contribute to this perception. Because the program combines different methodologies and techniques, it provides a rich training environment for students.

理解语言是我们日常生活中至关重要的一部分，用于社交、情感和信息目的。虽然听觉在感知语音信息中起着重要的作用，但视觉也是这一过程中不可或缺的一部分。多年来，人们已经认识到，在谈话中看别人的脸可以提高我们的语言理解能力。因此，语言感知，在其最自然的形式（面对面），是视觉和听觉。这笔拨款将支持一项研究计划，旨在了解语音的视觉和听觉语言方面在口语感知过程中是如何联系在一起的。**具体来说，在这个资助期间，我和我的学生将使用三种方法来探索涉及视听语音感知的感知和神经机制。第一种是研究视听语音整合发生的时间窗口。自从有声电影发明以来，研究人员和电影制作人已经知道，视频和声音之间严格的时间重叠绝不是人们将它们视为单一事件的必要条件。换句话说，人们无法在很大的时间偏移范围内判断轨迹是否不同步。我们将探索这种现象，作为一种方法来理解神经系统如何结合来自不同感官的信息。第二部分探讨凝视在视听语音感知中的作用。主动视觉决定了面部的哪些部分被选择性地处理，这些研究解决了这种选择如何影响语言感知。具体来说，我们想要确定信息收集（凝视行为）是如何受到模态间时间同步的影响的。最后一个研究领域涉及使用功能磁共振成像（fMRI）和脑电图（EEG）来研究视听语音的神经表征。在研究计划的所有组成部分中使用的语音刺激都是使用创新的动画和基于计算机的编辑程序创建的。**整个研究计划将对我们理解语音信息处理和实现这一非凡感知技能的神经机制做出重大贡献。近年来，我们使用可控的动画刺激来确定视听整合所需的时间和空间条件。通过功能磁共振成像和对神经系统患者的研究，我们已经开始确定导致这种感知的神经结构的复杂网络。由于该计划结合了不同的方法和技术，它为学生提供了丰富的培训环境。