STIMULATE: A Unified Framework for Multimodal Conversational Behaviors in Interactive Humanoid Agents

点击翻译按钮获取中文摘要

Humans communicate using speech, prosodic cues, hand gestures, gaze and facial expression. The focus of this project is the interaction among these modalities in humans and in interactive humanoid communicating agents. The goal is to be able to synthesize and understand natural face-to-face conversational behavior -- that is, spontaneous gesture and facial movements in the context of speech with intonation. Two converging lines of research will be carried out: first, a series of empirical experiments on the interaction among these modalities, including experiments on the association between gestural features and motion verbs, the relationship between turn-taking and information structure, the interaction between facial movements and gesture, and the distribution of listener-looking-at-speaker- gesture. Second, the continued development of an interactive humanoid agent, including generating gestural features in association with dialogue, refining dialogue generation to take into account turn-taking, distributing conversational load across speech, gesture, and facial expression, and replacing the current system of gesture input by dataglove with a stereo vision system. The results of this research will contribute to theory of the interaction between verbal and nonverbal modalities in human interaction, and aid in the development of autonomous agents and humanoid interfaces that depend on natural human communicative behavior for increased efficiency, naturalness and flexibility of human - computer interaction.