KDI: Segmental and Prosodic Optical Phonetics for Human and Machine Speech Processing

这是一个为期三年的标准奖项。当人们说话时，他们同时产生声学和光学语音信号。当听力和视觉正常的人面对面交流时，他们会利用这两种信号。能够看到和听到说话者可以在嘈杂或难以理解的语音条件下提高语音理解能力。这个多学科项目的目标是定量地描述光学语音信号，研究光学语音特征与声学和生理语音特征的关系，研究人类视觉语音感知的几个基本问题，并将所获得的知识应用于光学语音合成。在整个项目中，我们解决的主要问题是：(1)感知者可以从看到的说话者那里获得什么语音信息？(2)光学和声学信号与潜在的语音清晰度是如何关联的？(3)视觉语音感知的知觉和神经生理学基础是什么？以及(4)我们能证明这一知识对人工合成说话面孔的有用性吗？正在为这个项目记录一个多人说话的数据库。记录包括声学、光学(面部有后向反射器标记)和生理信号(电磁矢状面正中动脉造影术-EMA)。对最近文献中显示声学和光学语音测量以及外部(光学)和内部(生理)语音测量之间高度相关性的研究进行了后续研究。研究包括感知实验，以确定分段和韵律语音特征。我们正在使用电生理学方法研究聋人和听力成年人视觉言语知觉的神经生理学基础。光学语音合成正被用来(1)测试我们对控制音素和韵律视觉感知的线索的理解，以及(2)研究人类对光学语音特征敏感的神经生理学基础。该项目将对语音合成和视听自动语音识别领域的工程产生影响。它将扩大对人类言语感知及其在聋人和听力人中的神经生理学基础的理解。该项目的应用将包括第二语言培训、在环境噪声条件下提高语音传输质量和识别精度、高效存储和传输光学语音信息、视听感知实验中的刺激控制以及为听障人士加强沟通。主要研究人员组成的多学科团队代表了认知科学、言语知觉、语言学、电子工程和神经生理学等领域。

This is a three-year standard award. When people talk, they produce both acoustic and optical speech signals. When people with normal hearing and vision communicate face to face, they make use of both types of signals. Being able to see and hear a talker can enhance speech understanding in conditions of noisy or difficult-to-comprehend speech. The goals of this multidisciplinary project are to quantitatively characterize optical speech signals, examine how optical speech characteristics relate to acoustic and to physiologic speech characteristics, study several fundamental issues in human visual speech perception, and apply obtained knowledge to optical speech synthesis. Across the entire project, the main questions we address are: (1) What speech information can perceivers get from seeing talkers? (2) How are optical and acoustic signals related to underlying speech articulations? (3) What are the perceptual and neurophysiological bases for visual speech perception? and (4) Can we demonstrate usefulness of this knowledge for developing synthesis of artificial talking faces? A multi-talker database is being recorded for this project. Recordings include acoustic, optical (with retroreflector labeling on the faces), and physiologic signals (Electromagnetic Midsaggital Articulography -- EMA). Studies follow up on recent results in the literature showing high correlations between acoustic and optical speech measures, and between external (optical) and internal (physiologic) speech measures. Studies include perceptual experiments to determine segmental and prosodic speech characteristics. We are investigating the neurophysiologic bases for visual speech perception in deaf and hearing adults using electrophysiologic measures. Optical speech synthesis is being employed to (1) test our understanding of the cues that control visual perception of phonemes and prosody, and (2) investigate the neurophysiological bases for human sensitivity to optical speech characteristics. The project will impact engineering in the areas of speech synthesis and audiovisual automatic speech recognition. It will extend understanding of human speech perception and its neurophysiologic bases in deaf and hearing individuals. The applications that will derive from the project include ones in second language training, enhancement of speech transmission quality and recognition accuracy under conditions of environmental noise, efficient storage and transmission of optical speech information, stimulus control in audiovisual perceptual experiments, and communication enhancement for hearing impaired people. The multidisciplinary team of principal investigators represent the fields of cognitive science, speech perception, linguistics, electrical engineering, and neurophysiology.