Visual Speech Perception and Neural Processing

视觉语音感知和神经处理

• 批准号: 7322787
• 负责人: LYNNE Esther Bernstein
• 金额: $ 10万
• 依托单位: HOUSE RESEARCH INSTITUTE
• 依托单位国家: 美国
• 财政年份: 2007
• 资助国家: 美国
• 起止时间: 2007-02-01 至 2007-07-14
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/7322787
• 关键词: 3-Dimensional; Adult; Affect; Age; Auditory; Auditory Perception; Auditory area; Brain; Categories; Characteristics; Child; Complex; Condition; Data; Development; Discrimination; Electroencephalography; Electrophysiology (science); Event-Related Potentials; Face; Goals; Hearing; Hearing Impaired Persons; Individual; Information and Communication Theory; Intelligence; Knowledge; Lead; Learning; Least-Squares Analysis; Lipreading; Localized; Location; Measures; Methods; Neighborhoods; Numbers; Optics; Outcome Measure; Participant; Phonetics; Population; Process; Public Health; Questionnaires; Range; Reaction Time; Research; Response Latencies; Signal Transduction; Source; Speech; Speech Acoustics; Speech Perception; Speech Synthesizers; Speed; Stimulus; Structure; System; Testing; Training; Video Recording; Vision; Visual; Vocabulary; clinical application; density; design; early onset; extrastriate; extrastriate visual cortex; hearing impairment; paired stimuli; reconstruction; relating to nervous system; remediation; research study; speech processing; stimulus processing; visual stimulus

The talking face is very frequently characterized as a natural but ancillary source of speech information during face-to-face conversation. But typically for individuals with profound hearing impairments, auditory speech information is the ancillary source. Fortunately, many prelingually deaf adults, who rely on visual speech information for communication, demonstrate enhanced visual speech perception. Some hearing adults are also capable visual speech perceivers. Little is known about visual speech perception (lipreading). An important characteristic of the visual speech signal is that it conveys less phonetic information than does the acoustic speech signal, yet it affords adequate phonetic information to recognize, a high percentage of words.Our previous research suggests that the perceptual similarity structure of visual phonetic information is isomorphic with the physical similarity structure of visual stimuli. We hypothesize that expert lipreading is the result of sensitivity to small phonetic differences in visual speech stimuli, and that expert deaf lipreaders are more sensitive than hearing lipreaders to the phonetic information that renders phonemes perceptually dissimilar. This hypothesis will be tested in discrimination experiments designed to exploit the visual perceptual similarity of natural spoken syllables. However, natural speech stimuli are extremely complex, and we will also use our new visual speech synthesizer to investigate whether the physical data we used to characterize the speech stimulus space are an adequate representation of the information in the natural visual speech stimuli. Finally, our longterm goal is to determine where and when in the cortical hierarchy visual speech stimuli are processed for their phonetic information. To approach this longterm goal, an electrophysiology study will use what is learned about visual speech dissimilarity to localize visual phonetic processing temporally and spatially in cortex. Relevance to public health: Numbers of individuals with hearing loss increase as the population ages. Vision is a major source of speech input whenever auditory perception is impaired. Understanding the mechanisms that enhnace visual speech perception could lead to more effective and efficient clinical applications, such as evaluating speech perception and providing remediation to individuals with hearing impairment. Our visual speech synthesizer will be useful for training deaf children in vocabulary development and lipreading.

说话的脸经常被描述为一种自然但辅助的语音信息源 在面对面的交谈中。但对于有严重听力障碍的人来说， 语音信息是辅助源。幸运的是，许多依赖视觉的语前聋成年人， 用于通信的语音信息，表现出增强的视觉语音感知。一些听力 成年人也是有能力的视觉语言感知者。关于视觉言语感知（唇读）知之甚少。 视觉语音信号的一个重要特征是它比普通语音信号传达更少的语音信息。 声学语音信号，但它提供了足够的语音信息来识别， 我们以前的研究表明，视觉语音信息的知觉相似性结构 与视觉刺激的物理相似性结构同构。我们假设专家唇读是 对视觉言语刺激中微小语音差异敏感性的结果，以及聋唇读专家 比听唇读更敏感的语音信息呈现音素感知 不一样这一假设将在辨别实验中得到检验，该实验旨在利用视觉信号， 自然发音音节的感知相似性。然而，自然语音刺激是极其复杂的， 我们还将使用我们新的视觉语音合成器来调查我们用来 表征语音刺激空间是自然语言中信息的充分表示。 视觉言语刺激最后，我们的长期目标是确定在皮层层级中， 处理视觉语音刺激以获得它们的语音信息。为了实现这一长期目标， 电生理学研究将使用关于视觉语音相异性的知识来定位视觉语音 在皮层中进行时间和空间处理。与公共卫生的相关性： 听力损失随着人口老龄化而增加。视觉是语音输入的主要来源， 感知受损。理解增强视觉言语感知的机制可能导致 更有效和高效的临床应用，如评估言语感知和提供 对听力障碍者进行补救。我们的视觉语音合成器将有助于培训 聋儿的词汇发展和唇读能力。

项目成果

Visual phonetic processing localized using speech and nonspeech face gestures in video and point-light displays.

• DOI: 10.1002/hbm.21139
• 发表时间: 2011-10
• 期刊: HUMAN BRAIN MAPPING
• 影响因子: 4.8
• 作者: Bernstein, Lynne E.;Jiang, Jintao;Pantazis, Dimitrios;Lu, Zhong-Lin;Joshi, Anand
• 通讯作者: Joshi, Anand