STIMULATE: A Unified Framework for Multimodal Conversational Behaviors in Interactive Humanoid Agents

人类通过语言、韵律线索、手势、凝视和面部表情进行交流。这个项目的重点是这些模式在人类和交互类人通信代理之间的相互作用。我们的目标是能够综合和理解自然的面对面的对话行为——也就是说，在有语调的讲话背景下，自发的手势和面部动作。研究方向将集中在两个方面：首先，对这些模式之间的相互作用进行一系列实证实验，包括手势特征与动作动词的关联实验、轮转与信息结构的关系实验、面部动作与手势的相互作用实验、听者-看说话者-手势的分布实验。其次，交互式类人智能体的持续发展，包括生成与对话相关的手势特征，改进对话生成以考虑轮流，在语音、手势和面部表情之间分配会话负载，以及用立体视觉系统取代目前由dataglove输入手势的系统。本研究的结果将有助于建立人类互动中语言和非语言模式之间的互动理论，并有助于开发依赖于人类自然交流行为的自主代理和类人界面，以提高人机交互的效率、自然性和灵活性。

Humans communicate using speech, prosodic cues, hand gestures, gaze and facial expression. The focus of this project is the interaction among these modalities in humans and in interactive humanoid communicating agents. The goal is to be able to synthesize and understand natural face-to-face conversational behavior -- that is, spontaneous gesture and facial movements in the context of speech with intonation. Two converging lines of research will be carried out: first, a series of empirical experiments on the interaction among these modalities, including experiments on the association between gestural features and motion verbs, the relationship between turn-taking and information structure, the interaction between facial movements and gesture, and the distribution of listener-looking-at-speaker- gesture. Second, the continued development of an interactive humanoid agent, including generating gestural features in association with dialogue, refining dialogue generation to take into account turn-taking, distributing conversational load across speech, gesture, and facial expression, and replacing the current system of gesture input by dataglove with a stereo vision system. The results of this research will contribute to theory of the interaction between verbal and nonverbal modalities in human interaction, and aid in the development of autonomous agents and humanoid interfaces that depend on natural human communicative behavior for increased efficiency, naturalness and flexibility of human - computer interaction.