A multi-modal sensor fusion architecture for audio-visual speech understanding

用于视听语音理解的多模态传感器融合架构

• 批准号: 184129-2007
• 负责人: Basir, Otman
• 金额: $ 1.97万
• 依托单位: University of Waterloo
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2007
• 资助国家: 加拿大
• 起止时间: 2007-01-01 至 2008-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=363258
• 关键词: multi; modal; sensor; fusion; architecture

Modern speech recognition systems remain challenged in noisy environments despite all the technical advances that have been made to enhance their capabilities. There has been significant research work to address the effect of noise and word ambiguities on recognition performance. For speech recognition systems to be of practical use in noisy environments, such as in automobile, crowded areas and/or simultaneous human computer discourse applications, the issue of robustness must be addressed. The applicant maintains that this can possibly be accomplished by utilizing other sensing modalities to complement the acoustic signal of the speech. An example in line with this strategy is to fuse visual lip movements and expressions with the acoustic signal of the speech so as to maximize information gathering about the words uttered and to minimize the impact of acoustic noise.Understanding speech from visual information is an attractive approach that has captured the interest of many researchers to improve speech recognition performance. Not only does this approach have the potential to improve speech recognition performance but it may also be possible for speech impaired people to interact with devices in human machine interfacing applications.In this research project the applicant investigates the integration of audio and visual lip movement features (during speech signal production) to detect and recognize spoken phrases. A multi-modal fusion system will be designed. The system should be able to acquire visual and acoustic signals of speech and fuse them to detect and recognize words spoken by the speaker in a noisy environment. The system will also be tuned and tested in situations where the speaker is speech impaired. It is expected that the proposed research project will result in the training of 6 highly qualified personnel in human machine interfacing- an area of strategic importance to the Canadian economy.

尽管已经取得了所有技术进步来增强现代语音识别系统的功能，但现代语音识别系统在噪声环境中仍然面临挑战。人们已经开展了大量的研究工作来解决噪声和单词歧义对识别性能的影响。为了使语音识别系统在噪声环境中得到实际应用，例如在汽车、拥挤区域和/或同时人机对话应用中，必须解决鲁棒性问题。申请人认为，这可以通过利用其他感测方式来补充语音的声学信号来实现。符合这一策略的一个例子是将视觉嘴唇运动和表情与语音的声学信号融合起来，以便最大限度地收集有关所说单词的信息，并最大限度地减少声学噪声的影响。从视觉信息中理解语音是一种很有吸引力的方法，它引起了许多研究人员提高语音识别性能的兴趣。这种方法不仅具有提高语音识别性能的潜力，而且还可以让有语言障碍的人与人机接口应用中的设备进行交互。在这个研究项目中，申请人研究了音频和视觉嘴唇运动特征（在语音信号产生期间）的集成，以检测和识别口语短语。将设计多模态融合系统。该系统应该能够获取语音的视觉和听觉信号，并将它们融合起来，以检测和识别说话者在嘈杂环境中说出的单词。该系统还将在讲话者有言语障碍的情况下进行调整和测试。预计拟议的研究项目将培训 6 名人机界面方面的高素质人才，该领域对加拿大经济具有战略重要性。