RI: Small: Extending Verb Semantics with Causality towards Physical World

随着新一代认知机器人的出现，使用自然语言与这些机器人进行交流的能力变得越来越重要。言语交流经常涉及动词的使用，例如，要求机器人执行一些任务或监控一些身体活动。具体动作动词通常表示某一动作导致的状态变化；例如，“切一块披萨”的意思是对象披萨的状态会从一块变成几块。状态的变化可以通过不同的传感器从物理世界中感知到。给定人类的话语，如果机器人能够预测到动词所表示的状态的潜在变化，那么它就可以主动地感知环境，并更好地将语言与感知的物理世界联系起来，例如谁执行了动作，以及涉及哪些对象和位置。这种改进的连接将使许多依赖于人与机器人通信的应用程序受益。通过认知机器人，该项目将为K-12学生带来新的教育体验，并鼓励更广泛地参与工程。这项研究项目为具体动作动词开发了新的因果关系模型，以捕捉物理世界状态的预期变化。它根据具体动词可能如何改变环境(即因果关系)来增加具体动词的意义，并基于具体名词可能如何被行为改变(即启示)来增加具体名词的意义。它将因果关系模型结合到学习和推理算法中，以使语言与物理世界相联系。这项工作将提供一个新的维度，将动词语义与感知和行动联系起来。动词因果模型将允许机器人根据人类的语言话语预测潜在的状态变化。这一预测将提供自上而下的信息来指导视觉处理和动作建模。

With the emergence of a new generation of cognitive robots, the capability to communicate with these robots using natural language has become increasingly important. Verbal communication often involves the use of verbs, for example, to ask a robot to perform some tasks or to monitor some physical activities. Concrete action verbs often denote some change of state as a result of an action; for example, "slice a pizza" implies the state of the object pizza will be changed from one piece to several smaller pieces. The change of state can be perceived from the physical world through different sensors. Given a human utterance, if the robot can anticipate the potential change of the state signaled by the verbs, it can then actively sense the environment and better connect language with the perceived physical world such as who performs the action and what objects and locations are involved. This improved connection will benefit many applications relying on human-robot communication. Through a cognitive robot, this project will bring new educational experiences to K-12 students and encourage broader participation in engineering. This research project develops novel causality models for concrete action verbs to capture intended change of state of the physical world. It augments meanings of concrete verbs based on how they might change the environment (i.e., causality) and meanings of concrete nouns based on how they might be changed by actions (i.e., affordance). It incorporates causality models into learning and inference algorithms for grounding language to the physical world. This work will provide a new dimension to connect verb semantics to perception and action. Verb causality models will allow the robot to predict potential change of state from human linguistic utterances. This prediction will provide top-down information to guide visual processing and action modeling.