RI: Small: Extending Verb Semantics with Causality towards Physical World

随着新一代认知机器人的出现，使用自然语言与这些机器人进行交流的能力变得越来越重要。 口头交流通常涉及动词的使用，例如，要求机器人执行某些任务或监控某些身体活动。具体的动作动词通常表示动作所导致的某种状态变化;例如，“切片比萨饼”意味着比萨饼的状态将从一块变成几块。状态的变化可以通过不同的传感器从物理世界中感知。给定一个人类的话语，如果机器人可以预测动词所表示的状态的潜在变化，那么它就可以主动感知环境，并更好地将语言与感知的物理世界联系起来，例如谁执行动作以及涉及什么对象和位置。 这种改进的连接将使许多依赖于人机通信的应用程序受益。 通过认知机器人，该项目将为K-12学生带来新的教育体验，并鼓励更广泛地参与工程。 本研究计划为具体动作动词开发新的因果关系模型，以捕捉物理世界状态的预期变化。它根据具体动词如何改变环境来增加它们的含义（即，因果关系）和基于它们可能如何被动作改变的具体名词的含义（即，启示）。 它将因果关系模型纳入学习和推理算法中，以将语言与物理世界联系起来。 这项工作将提供一个新的维度连接动词语义的感知和行动。 动词因果关系模型将允许机器人从人类语言话语中预测潜在的状态变化。这种预测将提供自上而下的信息来指导视觉处理和动作建模。

With the emergence of a new generation of cognitive robots, the capability to communicate with these robots using natural language has become increasingly important. Verbal communication often involves the use of verbs, for example, to ask a robot to perform some tasks or to monitor some physical activities. Concrete action verbs often denote some change of state as a result of an action; for example, "slice a pizza" implies the state of the object pizza will be changed from one piece to several smaller pieces. The change of state can be perceived from the physical world through different sensors. Given a human utterance, if the robot can anticipate the potential change of the state signaled by the verbs, it can then actively sense the environment and better connect language with the perceived physical world such as who performs the action and what objects and locations are involved. This improved connection will benefit many applications relying on human-robot communication. Through a cognitive robot, this project will bring new educational experiences to K-12 students and encourage broader participation in engineering. This research project develops novel causality models for concrete action verbs to capture intended change of state of the physical world. It augments meanings of concrete verbs based on how they might change the environment (i.e., causality) and meanings of concrete nouns based on how they might be changed by actions (i.e., affordance). It incorporates causality models into learning and inference algorithms for grounding language to the physical world. This work will provide a new dimension to connect verb semantics to perception and action. Verb causality models will allow the robot to predict potential change of state from human linguistic utterances. This prediction will provide top-down information to guide visual processing and action modeling.