NRI: FND: Improving Robot Learning from Feedback and Demonstration using Natural Language

在从家庭到工作场所的广泛范围内部署通用机器人需要一个更可持续的模型来快速和强大地训练它们在未知环境中执行新任务，而无需机器人专家的干预。为了实现这一目标，已经探索了各种方法，允许普通人类用户使用各种形式的指导和交互来训练机器人，特别是通过在机器人学习执行任务时提供评估反馈，或者通过明确演示如何执行任务。当一个人向另一个人提供反馈或演示任务时，他们通常会用自然语言描述他们正在做的事情，为他们的评估或行动提供背景、澄清和/或解释。因此，该项目侧重于开发新的计算方法，使机器人能够更有效地从反馈和演示中学习，并利用伴随的自然语言叙述作为上下文。该项目开发了两种新的方法，通过整合语言基础、深度学习解释和从基本原理中学习的想法，使用语言来帮助交互式任务学习。第一种方法使用语言叙述作为一种“监督注意”的形式，将学习集中在环境的相关特征上，从而允许从有限的训练数据中进行有效的学习。首先，系统学习在机器人的感知中建立自然语言基础，利用之前在自动视频字幕和多模态语言基础方面的工作。接下来，使用最新的方法将人类语言叙述翻译成感知场上的显著性图，以直观地解释产生的语言基础网络的处理过程。最后，该显著性图用于监督从反馈和/或演示中学习的深度强化学习系统的注意机制，使其能够从有限的交互中更快、更有效地学习。第二种方法使用自然语言叙述来执行奖励塑造。在这种方法中，自然语言指令被映射到中间奖励，这可以无缝地集成到任何标准的强化学习算法中，再次提高学习的速度和准确性。这两种方法都是通过实验来评估的，使用它们来学习新任务，并定量地比较有和没有语言叙述的学习速度和有效性。假设语言叙述的使用将提高学习的速度和有效性。任务将包括使用视频游戏的模拟任务，通常用于评估强化学习，以及涉及导航和物体操作的现实世界机器人任务。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Deploying general purpose robots on a wide scale ranging from the home to the workplace requires a more sustainable model to quickly and robustly train them to perform novel tasks in unknown environments without the intervention of robotics experts. Toward this goal, various approaches have been explored to allow an ordinary human user to train a robot using various forms of instruction and interaction, specifically by providing evaluative feedback while a robot is learning to perform a task, or by explicitly demonstrating how to perform the task. When a person is providing feedback or demonstrating a task for another human, they typically describe what they are doing in natural language, providing context, clarification, and/or explanations for their evaluations or actions. Therefore, this project focuses on developing new computational methods that will enable robots to more efficiently and robustly learn from feedback and demonstration by leveraging accompanying natural language narration as context.The project develops two new approaches to using language to aid interactive task learning by integrating ideas from language grounding, explanation for deep learning, and learning from rationales. The first approach uses language narration as a form of "supervised attention" that focuses learning on relevant features of the environment, thereby allowing effective learning from limited training data. First, the system learns to ground natural language in the robot's perceptions, utilizing prior work on automated video captioning and multi-modal linguistic grounding. Next, human linguistic narration is translated to a saliency map over the perceptual field using recent methods for visually explaining the processing of the resulting language-grounding networks. Finally, this saliency map is used to supervise the attention mechanism of a deep-reinforcement learning system that learns from feedback and/or demonstration, allowing it to learn faster and more effectively from limited interaction. The second approach uses natural language narrations to perform reward shaping. In this approach, natural language instructions are mapped to intermediate rewards, which can be seamlessly integrated into any standard reinforcement learning algorithm, again improving the speed and accuracy of learning. Both of these approaches are experimentally evaluated by using them to learn new tasks and quantitatively comparing the speed and effectiveness of learning with and without linguistic narration. The hypothesis is that the use of linguistic narration will improve the speed and effectiveness of learning. Tasks will include simulated ones employing video games typically used to evaluate reinforcement learning and real-world robot tasks involving navigation and object manipulation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.