A multi-modal sensor fusion architecture for audio-visual speech understanding

点击翻译按钮获取中文摘要

Modern speech recognition systems remain challenged in noisy environments despite all the technical advances that have been made to enhance their capabilities. There has been significant research work to address the effect of noise and word ambiguities on recognition performance. For speech recognition systems to be of practical use in noisy environments, such as in automobile, crowded areas and/or simultaneous human computer discourse applications, the issue of robustness must be addressed. The applicant maintains that this can possibly be accomplished by utilizing other sensing modalities to complement the acoustic signal of the speech. An example in line with this strategy is to fuse visual lip movements and expressions with the acoustic signal of the speech so as to maximize information gathering about the words uttered and to minimize the impact of acoustic noise.