An Implementation of Multimedia Communication in Knowledge Acquisition Process

点击翻译按钮获取中文摘要

This research intends to implement a multimedia communication system for problem solving and knowledge acquisition, focusing on the first grade mathematics.1.Dialogue-based problem solving system.We have developed a problem solving system based on co-reference between drill texts of the first grade mathematics and dialogue with a tescher. It has been designed with emphasis on the issue in what situation of problem solving process it should utter, what sentence it should utter, and how it should interpret the response using contextual information. The constituents of the context are surface and case strusture of utterances, intention and sttention of the speaker, situation of dialogue, and world knowledge.2.Cooperative understanding of utterances and gestures.The teacher in general utters a sentence while he/she makes such a gesture as pointing to some objects in the text, or drawing some pictures. We have proposed a method of cooperative understanding of utterances and gestures in dialogue. The point of interest in the gesture is the movement of teacher's fingertip or pen. The location of the tip of the pen is detected from moving imades. Then the relationship between the features extracted from the moving points and linguistic information in the utterance as well as interpreting procedure of the gesture were presented.3.Knowledge acquisition, speech processing and face recognition.First, we made the assumption that knowledge acquired from dialogue is heuristics for problem solving, and presented a method of knowledge generalization and reuse. Then we confirmed that the speech interface can be used for dialogue-based problem solving process by experiment of speech recognition. Next, facial expression is important information in multimodal dialogue. We have proposed a method of detecting face elements such as eyes, nose and mouth. It uses local autocorrelation features, discriminant analysis, and genetic algorithm.