A multi-modal sensor fusion architecture for audio-visual speech understanding

现代语音识别系统在嘈杂的环境中仍然面临挑战，尽管已经取得了所有的技术进步来增强其能力。已经有大量的研究工作来解决噪声和单词歧义对识别性能的影响。为了使语音识别系统在噪声环境中具有实际用途，例如在汽车、拥挤区域和/或同时的人类计算机话语应用中，必须解决稳健性问题。申请人坚持认为，这可能通过利用其他感测模式来补充语音的声学信号来实现。一个符合该策略的例子是将视觉唇部动作和表情与语音的声学信号融合，以最大限度地收集关于所说单词的信息，并将声学噪声的影响降至最低。从视觉信息中理解语音是一种吸引人的方法，已经引起了许多研究人员的兴趣，以提高语音识别性能。这种方法不仅具有改善语音识别性能的潜力，而且对于语音受损的人来说也可能在人机接口应用中与设备交互。在该研究项目中，申请人调查了(在语音信号产生期间)音频和视觉唇部运动特征的集成以检测和识别所说的短语。将设计一个多通道融合系统。该系统应该能够采集语音的视觉和听觉信号，并对它们进行融合，以检测和识别说话人在噪声环境中所说的话。该系统还将在说话者言语受损的情况下进行调整和测试。预计拟议的研究项目将培训6000名高素质的人机界面人员，这是对加拿大经济具有战略重要性的领域。

Modern speech recognition systems remain challenged in noisy environments despite all the technical advances that have been made to enhance their capabilities. There has been significant research work to address the effect of noise and word ambiguities on recognition performance. For speech recognition systems to be of practical use in noisy environments, such as in automobile, crowded areas and/or simultaneous human computer discourse applications, the issue of robustness must be addressed. The applicant maintains that this can possibly be accomplished by utilizing other sensing modalities to complement the acoustic signal of the speech. An example in line with this strategy is to fuse visual lip movements and expressions with the acoustic signal of the speech so as to maximize information gathering about the words uttered and to minimize the impact of acoustic noise.Understanding speech from visual information is an attractive approach that has captured the interest of many researchers to improve speech recognition performance. Not only does this approach have the potential to improve speech recognition performance but it may also be possible for speech impaired people to interact with devices in human machine interfacing applications.In this research project the applicant investigates the integration of audio and visual lip movement features (during speech signal production) to detect and recognize spoken phrases. A multi-modal fusion system will be designed. The system should be able to acquire visual and acoustic signals of speech and fuse them to detect and recognize words spoken by the speaker in a noisy environment. The system will also be tuned and tested in situations where the speaker is speech impaired. It is expected that the proposed research project will result in the training of 6 highly qualified personnel in human machine interfacing- an area of strategic importance to the Canadian economy.