Studies on Multimodal Communication by Integrating Speech and Diagram

点击翻译按钮获取中文摘要

The aim of this research is to realize multimodal communication between human and machines through implementing multimodal drawing system with speech input. The results of this research are below :1. Integration of speech, gesture and diagramWe developed multi-mode integration method without assuming the synchronization of each input elements. The validity of this integration method was examined by evaluating the improvement of accuracy of information input in multimodal dictation system.2. Implementation of multimodal interface using speech and pointing gestureWe developed the cooperation mechanism which uses the result of pointing gesture recognition, which is reliable channel, for the recognition of speech, which is relatively unreliable channel. By using the result of pointing gesture recognition, the accuracy of speech recognition raised about 6%. Using this integration method, we implemented a multimodal drawing system with speech input.3. Interpretation of utterance using diagrammatic and contextual information in multimodal drawing system We realized utterance understanding mechanism which uses context management mechanism by stack in order to deal with spontaneous speech (e. g. ellipsis, reference expression, etc.). We integrated this utterance understanding mechanism to multimodal drawing system.