Study on Integrated Processing of Speech and Gesture in Multimodal Communication

多模态通信中语音与手势综合处理研究

• 批准号: 10480083
• 负责人: SHIRAI Katsuhiko
• 金额: $ 5.89万
• 依托单位: Waseda University
• 依托单位国家: 日本
• 项目类别: Grant-in-Aid for Scientific Research (B).
• 财政年份: 1998
• 资助国家: 日本
• 起止时间: 1998 至 2000
• 项目状态: 已结题
• 来源: https://kaken.nii.ac.jp/en/grant/KAKENHI-PROJECT-10480083/
• 关键词: Multimodal Communication; Gesture Recognition; Speech Recognition; Partly-Hidden Markov Model; Multi-Person Conversation; Dialogue Control; Misunderstanding Detection; Domain Independent Platform; 複数話者対話; 統計的発話交代モデル; 部分空間法; 顔面像抽出; 複合周波数帯域型音声認識

The purpose of this research is to develop the multimodal communication system which can recognize multimodal Information such as speech and gesture on natural dialog, understand the intention of human by the integration of them, and respond to human appropriately.First of all, it is necessary to clarify the structure of understanding of human intention by the integration of multimodal information and response by multiple modalities. Therefore we have analyzed the acoustic features of speech such as fillers and the roles of gestures such as head movement on the various natural human dialogues.Then we have made studies of speech and gesture recognition algorithm that is fundamental technique for multimodal communication system. We suggest a recombination strategy for multi-band automatic speech recognition which gives more accurate recognition, especially in noisy acoustic environments. And we propose a speech decoder in which the language models are modified to deal with timing of the turn taking and the speaker models are also utilized. We apply a new pattern matching method, Partly-Hidden Markov model, in which the first state is hidden and the second one is observable, to gesture recognition. And we propose the face extraction and the pose detection method to recognize the head movement.Finally, we have implemented multimodal communication model to the human-machine dialogue system. This system uses a method of generalization considering trade-off between variety of dialogue and easiness to describes rules and provides a domain independent platform. Also, it has a spoken dialogue control model for improvement of dialogue efficiency and a dialogue management model for detection of misunderstanding in spoken dialogue system.

本研究的目的是开发一个能够在自然对话中识别语音、手势等多通道信息的多通道交流系统，通过它们的整合来理解人类的意图，并对人类做出恰当的反应。首先，需要通过整合多通道信息和多种通道的响应来明确人类意图的理解结构。因此，我们分析了填充词等语音的声学特征和头部运动等手势在各种自然人类对话中的作用。然后，我们研究了多模式通信系统的基础技术--语音和手势识别算法。我们提出了一种多频段自动语音识别的重组策略，它可以提供更准确的识别，特别是在噪声环境下。我们还提出了一种语音解码器，其中语言模型被修改来处理话轮转换的时间，同时也利用了说话人模型。我们将一种新的模式匹配方法--部分隐马尔可夫模型应用于手势识别，其中第一个状态是隐藏的，第二个状态是可观测的。提出了人脸提取和姿态检测方法来识别头部运动。最后，将多通道通信模型实现到人机对话系统中。该系统采用一种兼顾对话多样性和易用性的泛化方法来描述规则，提供了一个与领域无关的平台。此外，它还具有用于提高对话效率的口语对话控制模型和用于检测口语对话系统中误解的对话管理模型。

项目成果

K.Aoyama,K.Shirai: "Controlling Non-verbal Information in Speaker-change for Spoken Dialogue"2000 IEEE International Conference on Systems Man and Cybernetics (SMC2000). 1354-1359 (2000)

K.Aoyama，K.Shirai：“控制口语对话中说话人变化中的非语言信息”2000 年 IEEE 国际系统人与控制论会议 (SMC2000)。

Shigeki Okawa 他: "A Recombination Strategy for Multi-band Speech Recognition Based on Mutual Information Criterion"Proc. of EUROSPEECH'99. Vol.2. 603-606 (1999)

Shigeki Okawa 等人：“基于互信息准则的多频带语音识别重组策略”Proc. of EUROSPEECH99 第 2 卷 603-606（1999 年）

M.Yokoyama,K.Shirai: "Use of Non-Verbal Information in Communication between Human and Robot"Proc.of International Conference on Spoken Language Processing (ICSLP). 2351-2354 (1998)

M.Yokoyama，K.Shirai：“人与机器人交流中非语言信息的使用”国际口语处理会议（ICSLP）会议记录。

H.Kikuchi,K.Shirai: "Controlling Dialogue Strategy According to Performance of Processes"ESCA Workshop,Session5.2. 85-88 (1999)

H.Kikuchi、K.Shirai：“根据流程性能控制对话策略”ESCA 研讨会，Session5.2。

K.Aoyama,K.Shirai: "DESIGNING A DOMAIN INDEPENDENT PLATFORM OF SPOKEN DIALOGUE SYSTEM"Proc.of International Conference on Spoken Language Processing (ICSLP). (CD-ROM). (2000)

K.Aoyama，K.Shirai：“设计口语对话系统的领域独立平台”Proc.of 国际口语处理会议（ICSLP）。