ITR-Collaborative Research: Development and Evaluation of a Hybrid Concatenative/Rule-Based Visual Speech Synthesis System

这个项目的目标是开发一张合成的会说话的脸。早在人类被要求阅读由计算机呈现的印刷文本之前，他们就已经发展出感知和整合听觉和视觉(AV)语音信息的复杂能力。与只听说话者相比，看和听都能减少认知工作量，提高理解力。要实现AV语音用于人机交互的优势，需要合成可视语音，从而提供无限的可视语音图像，而不必预先记录数据。这里的方法是用语音声学来驱动光学语音合成。计算方法得到了从声学到光学的转换模型。该方法利用双音素捕获的语音产生协同发音信息来产生自然的视觉语音图像。该方法直接应用于自然声学语音特征，以获得声光信号之间的协调。合成的视觉语音基于纹理映射的线框模型。通过同时记录的3-D光学、音频和视频数据，获得了为合成奠定基础的自然语音语料库。综合开发是以人类知觉测试为指导的。将分发DVD存档语料库。该项目将扩大获取信息的机会，改善不同群体获取知识的情况，例如：仍在学习识字技能的儿童；识字能力不足的成年人；使用第二语言的人；以及依赖视听语言的听力受损的人。结果将通过专业渠道广泛传播。研究生和本科生将参加。

This project's goal is to develop a synthetic talking face. Humans developed sophisticated abilities to perceive and integrate auditory and visual (AV) speech information long before they were required to read printed text presented by computers. Seeing as well as hearing speech reduces the cognitive workload and improves comprehension over only hearing the talker. To realize the advantages of AV speech for human-computer interactions requires synthesizing visual speech, thereby providing an unlimited supply of visual speech images without having to pre-record data. The approach here is to drive optical speech synthesis with speech acoustics. Computational methods obtain models of the transformation from acoustics to optics. The method capitalizes on the speech production coarticulatory information captured by diphones to produce naturalistic visual speech images. The method is applied directly to natural acoustic speech features to obtain coordination between acoustic and optical signals. The synthesized visual speech is based on a texture-mapped wire frame model. A natural speech corpus to base the synthesis is being obtained via simultaneously recorded 3-D optical, audio, and video data. Synthesis development is guided by human perceptual testing. The DVD archived corpus will be disseminated. The project will lead to expanded access to information and improvement in obtaining knowledge by diverse groups of individuals, for example: children still acquiring literacy skills; adults with inadequate literacy; individuals who are using a second language; and individuals with hearing losses who rely on audiovisual speech. Results will be disseminated broadly through professional outlets. Graduate and undergraduate students will participate.