An Investigation of Cooperative Understanding of Utterances and Gestures Based on Interaction in Semantics Level

点击翻译按钮获取中文摘要

We are developing a problem solving and knowledge acquisition system based on co-reference between drill texts and dialogue with a teacher, focusing on first-grade mathematics. This research proposed a method of cooperative understanding of utterances and gestures.(1) Contextual information processing.We defined the context of dialogue, which consists of surface and case structure of utterances, intention and attention of the speaker, situation of dialogue, and world knowledge. We then presented the algorithms of generating utterances from the system as well as interpreting responses from the teacher using contextual information.(2) Analysis of gestures and utterances.Our point of interest is the movement of the tip of teacher's pen. We developed a simple input device to detect the three-dimensional coordinates of the tip of pen, and presented the algorithms to extract features from moving points. A feature-based approach is used for gesture recognition. We then proposed a method of parsing word candidates given from speech recognition program.(3) Cooperative understanding of utterances and gestures.We defined a multi-modal semantic representation to describe the meaning of utterances and gestures, and showed how to integrate our algorithms for utterance and gesture analysis. We concluded with an evaluation of the understanding system against the design principles, which provide the basis for the integration of multi-modal information during a dialogue.(4) Generation of gestures in cooperation with utterances.Gestures such as pointing of objects on a drill text or drawing of pictures, are represented by movement of a pen, and are displayed as three-dimensional graphical data. We defined a gesture frame and gesture element as an intermediate representation, and presented algorithms of generating them from the semantic representation with the synchronized phrase..