Study on Integrated Processing of Speech and Gesture in Multimodal Communication

本研究的目的是开发一个能够在自然对话中识别语音、手势等多通道信息的多通道交流系统，通过它们的整合来理解人类的意图，并对人类做出恰当的反应。首先，需要通过整合多通道信息和多种通道的响应来明确人类意图的理解结构。因此，我们分析了填充词等语音的声学特征和头部运动等手势在各种自然人类对话中的作用。然后，我们研究了多模式通信系统的基础技术--语音和手势识别算法。我们提出了一种多频段自动语音识别的重组策略，它可以提供更准确的识别，特别是在噪声环境下。我们还提出了一种语音解码器，其中语言模型被修改来处理话轮转换的时间，同时也利用了说话人模型。我们将一种新的模式匹配方法--部分隐马尔可夫模型应用于手势识别，其中第一个状态是隐藏的，第二个状态是可观测的。提出了人脸提取和姿态检测方法来识别头部运动。最后，将多通道通信模型实现到人机对话系统中。该系统采用一种兼顾对话多样性和易用性的泛化方法来描述规则，提供了一个与领域无关的平台。此外，它还具有用于提高对话效率的口语对话控制模型和用于检测口语对话系统中误解的对话管理模型。

The purpose of this research is to develop the multimodal communication system which can recognize multimodal Information such as speech and gesture on natural dialog, understand the intention of human by the integration of them, and respond to human appropriately.First of all, it is necessary to clarify the structure of understanding of human intention by the integration of multimodal information and response by multiple modalities. Therefore we have analyzed the acoustic features of speech such as fillers and the roles of gestures such as head movement on the various natural human dialogues.Then we have made studies of speech and gesture recognition algorithm that is fundamental technique for multimodal communication system. We suggest a recombination strategy for multi-band automatic speech recognition which gives more accurate recognition, especially in noisy acoustic environments. And we propose a speech decoder in which the language models are modified to deal with timing of the turn taking and the speaker models are also utilized. We apply a new pattern matching method, Partly-Hidden Markov model, in which the first state is hidden and the second one is observable, to gesture recognition. And we propose the face extraction and the pose detection method to recognize the head movement.Finally, we have implemented multimodal communication model to the human-machine dialogue system. This system uses a method of generalization considering trade-off between variety of dialogue and easiness to describes rules and provides a domain independent platform. Also, it has a spoken dialogue control model for improvement of dialogue efficiency and a dialogue management model for detection of misunderstanding in spoken dialogue system.