An Implementation of Multimedia Communication in Knowledge Acquisition Process

本研究以一年级数学为研究对象，设计并实现了一个用于问题解决和知识获取的多媒体交流系统。1.基于对话的问题解决系统。它的设计重点是在解决问题的过程中应该说什么，应该说什么句子，以及应该如何利用语境信息来解释回答。语境的构成要素是话语的表层和个案结构、说话人的意图和注意力、对话情境和世界知识。2.话语和手势的合作理解。教师一般是指着课文中的一些物体或画出一些图画时说出一句话。我们提出了一种对话中话语和手势的合作理解方法。这个手势的兴趣点是老师的指尖或笔的移动。通过移动笔尖来检测笔尖的位置。3.知识获取、语音处理和人脸识别。首先，我们假设从对话中获取的知识是问题求解的启发式知识，并提出了一种知识泛化和重用的方法。然后通过语音识别实验，证实了该语音界面可以用于基于对话的问题解决过程。其次，面部表情是多通道对话中的重要信息。我们提出了一种检测眼睛、鼻子和嘴巴等人脸元素的方法。它使用了局部自相关特征、判别分析和遗传算法。

This research intends to implement a multimedia communication system for problem solving and knowledge acquisition, focusing on the first grade mathematics.1.Dialogue-based problem solving system.We have developed a problem solving system based on co-reference between drill texts of the first grade mathematics and dialogue with a tescher. It has been designed with emphasis on the issue in what situation of problem solving process it should utter, what sentence it should utter, and how it should interpret the response using contextual information. The constituents of the context are surface and case strusture of utterances, intention and sttention of the speaker, situation of dialogue, and world knowledge.2.Cooperative understanding of utterances and gestures.The teacher in general utters a sentence while he/she makes such a gesture as pointing to some objects in the text, or drawing some pictures. We have proposed a method of cooperative understanding of utterances and gestures in dialogue. The point of interest in the gesture is the movement of teacher's fingertip or pen. The location of the tip of the pen is detected from moving imades. Then the relationship between the features extracted from the moving points and linguistic information in the utterance as well as interpreting procedure of the gesture were presented.3.Knowledge acquisition, speech processing and face recognition.First, we made the assumption that knowledge acquired from dialogue is heuristics for problem solving, and presented a method of knowledge generalization and reuse. Then we confirmed that the speech interface can be used for dialogue-based problem solving process by experiment of speech recognition. Next, facial expression is important information in multimodal dialogue. We have proposed a method of detecting face elements such as eyes, nose and mouth. It uses local autocorrelation features, discriminant analysis, and genetic algorithm.