A multi-modal sensor fusion architecture for audio-visual speech understanding

点击翻译按钮获取中文摘要

Modern speech recognition systems remain challenged in noisy environments despite all the technical advances that have been made to enhance their capabilities. There has been significant research work to address the effect of noise and word ambiguities on recognition performance. For speech recognition systems to be of practical use in noisy environments, such as in automobile, crowded areas and/or simultaneous human computer discourse applications, the issue of robustness must be addressed. The applicant maintains that this can possibly be accomplished by utilizing other sensing modalities to complement the acoustic signal of the speech. An example in line with this strategy is to fuse visual lip movements and expressions with the acoustic signal of the speech so as to maximize information gathering about the words uttered and to minimize the impact of acoustic noise.Understanding speech from visual information is an attractive approach that has captured the interest of many researchers to improve speech recognition performance. Not only does this approach have the potential to improve speech recognition performance but it may also be possible for speech impaired people to interact with devices in human machine interfacing applications.In this research project the applicant investigates the integration of audio and visual lip movement features (during speech signal production) to detect and recognize spoken phrases. A multi-modal fusion system will be designed. The system should be able to acquire visual and acoustic signals of speech and fuse them to detect and recognize words spoken by the speaker in a noisy environment. The system will also be tuned and tested in situations where the speaker is speech impaired. It is expected that the proposed research project will result in the training of 6 highly qualified personnel in human machine interfacing- an area of strategic importance to the Canadian economy.