KDI: Segmental and Prosodic Optical Phonetics for Human and Machine Speech Processing

这是一个三年的标准奖项。当人们说话时，他们会同时产生声音和光语音信号。当听力和视力正常的人面对面交流时，他们会使用这两种信号。能够看到和听到说话的人可以在嘈杂或难以理解的语言条件下提高对语言的理解。这个多学科项目的目标是定量表征光学语音信号，研究光学语音特征与声学和生理语音特征的关系，研究人类视觉语音感知的几个基本问题，并将获得的知识应用于光学语音合成。在整个项目中，我们解决的主要问题是：(1)感知者可以从看到的说话者那里得到什么语音信息？(2)光和声信号与潜在的语音发音有何关系？(3)视觉言语感知的知觉和神经生理基础是什么？(4)我们能否证明这些知识对开发人工说话面孔的合成有用？正在为这个项目记录一个多话务员数据库。记录包括声学、光学（面部有后向反射器标记）和生理信号（电磁矢状中关节成像- EMA）。最近的研究结果表明，声学和光学语音测量之间以及外部（光学）和内部（生理）语音测量之间存在高度相关性。研究包括知觉实验，以确定分段和韵律语音特征。我们正在用电生理学方法研究聋人和听力正常的成年人视觉语言感知的神经生理学基础。光学语音合成被用于(1)测试我们对控制音素和韵律视觉感知的线索的理解，以及(2)研究人类对光学语音特征敏感性的神经生理学基础。该项目将影响语音合成和视听自动语音识别领域的工程。它将扩展对人类语言感知及其在聋人和听力个体中的神经生理基础的理解。该项目将应用于第二语言训练、提高环境噪声条件下的语音传输质量和识别精度、光学语音信息的高效存储和传输、视听感知实验中的刺激控制、增强听障人士的沟通等领域。主要研究人员的多学科团队代表了认知科学、言语感知、语言学、电子工程和神经生理学等领域。

This is a three-year standard award. When people talk, they produce both acoustic and optical speech signals. When people with normal hearing and vision communicate face to face, they make use of both types of signals. Being able to see and hear a talker can enhance speech understanding in conditions of noisy or difficult-to-comprehend speech. The goals of this multidisciplinary project are to quantitatively characterize optical speech signals, examine how optical speech characteristics relate to acoustic and to physiologic speech characteristics, study several fundamental issues in human visual speech perception, and apply obtained knowledge to optical speech synthesis. Across the entire project, the main questions we address are: (1) What speech information can perceivers get from seeing talkers? (2) How are optical and acoustic signals related to underlying speech articulations? (3) What are the perceptual and neurophysiological bases for visual speech perception? and (4) Can we demonstrate usefulness of this knowledge for developing synthesis of artificial talking faces? A multi-talker database is being recorded for this project. Recordings include acoustic, optical (with retroreflector labeling on the faces), and physiologic signals (Electromagnetic Midsaggital Articulography -- EMA). Studies follow up on recent results in the literature showing high correlations between acoustic and optical speech measures, and between external (optical) and internal (physiologic) speech measures. Studies include perceptual experiments to determine segmental and prosodic speech characteristics. We are investigating the neurophysiologic bases for visual speech perception in deaf and hearing adults using electrophysiologic measures. Optical speech synthesis is being employed to (1) test our understanding of the cues that control visual perception of phonemes and prosody, and (2) investigate the neurophysiological bases for human sensitivity to optical speech characteristics. The project will impact engineering in the areas of speech synthesis and audiovisual automatic speech recognition. It will extend understanding of human speech perception and its neurophysiologic bases in deaf and hearing individuals. The applications that will derive from the project include ones in second language training, enhancement of speech transmission quality and recognition accuracy under conditions of environmental noise, efficient storage and transmission of optical speech information, stimulus control in audiovisual perceptual experiments, and communication enhancement for hearing impaired people. The multidisciplinary team of principal investigators represent the fields of cognitive science, speech perception, linguistics, electrical engineering, and neurophysiology.