权益分类	功能权益	普通用户	{{item.name}}会员
{{category.name}}	{{benefitItem.name}}

An Investigation of Cooperative Understanding of Utterances and Gestures Based on Interaction in Semantics Level

基于语义层面交互的言语和手势合作理解研究

基本信息

批准号：
10680388
负责人：
ENDO Tsutomu
金额：
$ 0.7万
依托单位：
Kyushu Institute of Technology (2000)Oita University (1998-1999)
依托单位国家：
日本
项目类别：
Grant-in-Aid for Scientific Research (C)
财政年份：
1998
资助国家：
日本
起止时间：
1998 至 2000
项目状态：
已结题

项目摘要

We are developing a problem solving and knowledge acquisition system based on co-reference between drill texts and dialogue with a teacher, focusing on first-grade mathematics. This research proposed a method of cooperative understanding of utterances and gestures.(1) Contextual information processing.We defined the context of dialogue, which consists of surface and case structure of utterances, intention and attention of the speaker, situation of dialogue, and world knowledge. We then presented the algorithms of generating utterances from the system as well as interpreting responses from the teacher using contextual information.(2) Analysis of gestures and utterances.Our point of interest is the movement of the tip of teacher's pen. We developed a simple input device to detect the three-dimensional coordinates of the tip of pen, and presented the algorithms to extract features from moving points. A feature-based approach is used for gesture recognition. We then proposed a method of parsing word candidates given from speech recognition program.(3) Cooperative understanding of utterances and gestures.We defined a multi-modal semantic representation to describe the meaning of utterances and gestures, and showed how to integrate our algorithms for utterance and gesture analysis. We concluded with an evaluation of the understanding system against the design principles, which provide the basis for the integration of multi-modal information during a dialogue.(4) Generation of gestures in cooperation with utterances.Gestures such as pointing of objects on a drill text or drawing of pictures, are represented by movement of a pen, and are displayed as three-dimensional graphical data. We defined a gesture frame and gesture element as an intermediate representation, and presented algorithms of generating them from the semantic representation with the synchronized phrase..

我们正在开发一个问题解决和知识获取系统，基于练习文本和与老师对话的共同参考，重点是一年级数学。本研究提出了一种合作理解话语和手势的方法。(1)语境信息处理。我们定义了对话的语境，包括话语的表层和格结构、说话人的意图和注意力、对话情境和世界知识。然后给出了从系统中生成话语以及利用上下文信息解释教师回答的算法。(2)手势和话语分析。我们的兴趣点是教师笔尖的移动。我们开发了一种简单的输入设备来检测笔尖的三维坐标，并给出了从运动点提取特征的算法。手势识别采用基于特征的方法。(3)话语和手势的协同理解，定义了一种多模式语义表示来描述话语和手势的含义，并给出了如何将我们的算法整合到话语和手势分析中。最后，我们根据设计原则对理解系统进行了评估，这些原则为对话过程中整合多通道信息提供了基础。(4)结合话语生成手势。手势的生成通过笔的移动来表示，并以三维图形数据的形式显示。我们定义了一个手势框架和手势元素作为中间表示，并给出了从同步短语的语义表示中生成它们的算法。