Studies on Multimodal Communication by Integrating Speech and Diagram

本研究的目的是通过实现语音输入的多模态绘图系统，实现人机之间的多模态交流。本研究的主要结果如下：1.语音、手势和手势的集成我们开发了多模式集成方法，而无需假设每个输入元素的同步。通过评价多模态听写系统中信息输入准确性的提高，检验了该集成方法的有效性.利用语音和指向手势实现多模态接口我们开发了协作机制，该机制使用指向手势识别的结果，这是可靠的通道，用于识别语音，这是相对不可靠的通道。利用手势识别的结果，语音识别的准确率提高了6%左右。利用这种集成方法，我们实现了一个语音输入的多通道绘图系统.多模态绘图系统中基于图形和上下文信息的话语理解我们通过堆栈实现了使用上下文管理机制的话语理解机制，以处理自发语音（例如：G.省略号、引用表达式等）。我们将这种话语理解机制集成到多模态绘图系统中。

The aim of this research is to realize multimodal communication between human and machines through implementing multimodal drawing system with speech input. The results of this research are below :1. Integration of speech, gesture and diagramWe developed multi-mode integration method without assuming the synchronization of each input elements. The validity of this integration method was examined by evaluating the improvement of accuracy of information input in multimodal dictation system.2. Implementation of multimodal interface using speech and pointing gestureWe developed the cooperation mechanism which uses the result of pointing gesture recognition, which is reliable channel, for the recognition of speech, which is relatively unreliable channel. By using the result of pointing gesture recognition, the accuracy of speech recognition raised about 6%. Using this integration method, we implemented a multimodal drawing system with speech input.3. Interpretation of utterance using diagrammatic and contextual information in multimodal drawing system We realized utterance understanding mechanism which uses context management mechanism by stack in order to deal with spontaneous speech (e. g. ellipsis, reference expression, etc.). We integrated this utterance understanding mechanism to multimodal drawing system.