RI: Small: Extending Verb Semantics with Causality towards Physical World

随着新一代认知机器人的出现，使用自然语言与这些机器人进行交流的能力变得越来越重要。口头交流通常涉及动词的使用，例如，要求机器人执行某些任务或监视某些身体活动。具体的动作动词通常表示由于某一动作而引起的某种状态的变化；例如，“slice a pizza”意味着对象pizza的状态将从一块变成几个更小的块。状态的变化可以通过不同的传感器从物理世界感知到。给定一个人的话语，如果机器人能够预测到动词所表示的状态的潜在变化，那么它就能主动感知环境，更好地将语言与感知到的物理世界联系起来，比如谁在执行动作，涉及到什么物体和位置。这种改进的连接将使许多依赖人机通信的应用受益。通过认知机器人，该项目将为K-12学生带来新的教育体验，并鼓励更广泛地参与工程。本研究项目为具体动作动词开发了新的因果关系模型，以捕捉物理世界状态的预期变化。它根据具体动词如何改变环境（即因果关系）来增加具体动词的意义，根据具体名词如何被行为改变（即功能）来增加具体名词的意义。它将因果关系模型整合到学习和推理算法中，以便将语言与物理世界联系起来。这项工作将为动词语义与感知和行为的联系提供一个新的维度。动词因果关系模型将允许机器人预测人类语言话语的潜在状态变化。这种预测将提供自上而下的信息来指导视觉处理和动作建模。

With the emergence of a new generation of cognitive robots, the capability to communicate with these robots using natural language has become increasingly important. Verbal communication often involves the use of verbs, for example, to ask a robot to perform some tasks or to monitor some physical activities. Concrete action verbs often denote some change of state as a result of an action; for example, "slice a pizza" implies the state of the object pizza will be changed from one piece to several smaller pieces. The change of state can be perceived from the physical world through different sensors. Given a human utterance, if the robot can anticipate the potential change of the state signaled by the verbs, it can then actively sense the environment and better connect language with the perceived physical world such as who performs the action and what objects and locations are involved. This improved connection will benefit many applications relying on human-robot communication. Through a cognitive robot, this project will bring new educational experiences to K-12 students and encourage broader participation in engineering. This research project develops novel causality models for concrete action verbs to capture intended change of state of the physical world. It augments meanings of concrete verbs based on how they might change the environment (i.e., causality) and meanings of concrete nouns based on how they might be changed by actions (i.e., affordance). It incorporates causality models into learning and inference algorithms for grounding language to the physical world. This work will provide a new dimension to connect verb semantics to perception and action. Verb causality models will allow the robot to predict potential change of state from human linguistic utterances. This prediction will provide top-down information to guide visual processing and action modeling.