Correcting robot behaviour through embodied natural language interaction

我的工作将在语义解析领域：即创建自动化系统的任务，将自然语言输入映射到捕获用户交流意图的可执行输出。例如，“把红块放在蓝块上”可以映射到一些正式的表示，比如“（Put, rb_1, bb_1）”，这可以由机器人执行。语义解析的一种常见方法是赋予系统有关领域中可能的操作和状态的完整知识：任务是学习如何使用自然语言来引用已知的领域概念和操作。通常，这些系统假设自然语言字符串要么是指示性的，描述当前视觉场景的某些方面，要么是命令性的，描述要执行的操作。但是，人类语言使用者表达了许多其他类型的与他们所体现的环境的连贯关系。例如，用户可以通过表达一些规则或启发式来“纠正”机器人当前正在做的事情，当这些规则或启发式被准确理解时，应该用于修改机器人将来执行此类动作的方式。例如，为了响应机器人将绿色块放在红色块上的动作，用户可能会说“不要将绿色块放在蓝色块上。”把红色方块放在蓝色方块上。”这将改变系统在蓝色块上放置块的未来操作。第一步是创建一个系统，它可以像上面描述的那样改变它的行为。然而，我们将对视觉和话语理解等感官输入做出许多假设，这样这些假设就可以被视为确定性的。一旦这个工作成功，我们将放弃对黄金标准解析和可视化处理的假设，并以概率方式对过程的噪声方面进行建模。

My work will be in the arena of semantic parsing: i.e., the task of creating automated systems that map natural language input into an executable output that captures the communicative intention of the user. For example, "Put the red block on the blue block" could map onto some formal representation, say "(put, rb_1, bb_1)", that can be executed by a robot. One common approach to semantic parsing is to endow the system with complete knowledge of the possible actions and states in the domain: the task is to learn how natural language is used to refer to known domain concepts and actions. Typically, these systems assume that the natural language string is either indicative and describes some aspect of the current visual scene, or it is imperative and it describes an action to be carried out. But human language users express many other kinds of coherent relations to their embodied environment. For example, a user might *correct* what a robot is currently doing, by expressing some rule or heuristic that, when accurately understood, should be used to revise the way the robot performs such actions in future. For example, in response to the robot performing an action where it is putting a green block on a red block,, the user might state "Don't put green blocks on blue blocks. Put red blocks on blue blocks". This should alter future actions where the system is putting a block on a blue block. The first step will be to create a system that changes its behaviour as described above. However, we will make many assumptions about the sensory input such as vision and understanding of discourse, such that these can be treated as deterministic. Once this works successfully, we will drop assumptions about gold standard parsing and visual processing, and model the noisy aspects of the process probabilistically.