A multi-modal sensor fusion architecture for audio-visual speech understanding

现代语音识别系统在嘈杂的环境中仍然面临挑战，尽管已经取得了所有的技术进步来增强其能力。已经有大量的研究工作来解决噪声和单词歧义对识别性能的影响。为了使语音识别系统在噪声环境中具有实际用途，例如在汽车、拥挤区域和/或同时的人类计算机话语应用中，必须解决稳健性问题。申请人坚持认为，这可能通过利用其他感测模式来补充语音的声学信号来实现。符合这一策略的一个例子是将视觉嘴唇动作和表情与语音的声音信号融合，以便最大限度地收集关于所发出的单词的信息，并将声音噪声的影响降至最低。

Modern speech recognition systems remain challenged in noisy environments despite all the technical advances that have been made to enhance their capabilities. There has been significant research work to address the effect of noise and word ambiguities on recognition performance. For speech recognition systems to be of practical use in noisy environments, such as in automobile, crowded areas and/or simultaneous human computer discourse applications, the issue of robustness must be addressed. The applicant maintains that this can possibly be accomplished by utilizing other sensing modalities to complement the acoustic signal of the speech. An example in line with this strategy is to fuse visual lip movements and expressions with the acoustic signal of the speech so as to maximize information gathering about the words uttered and to minimize the impact of acoustic noise.