Collaborative Research: HCC: Medium: Aligning Robot Representations with Humans

该项目旨在使机器人更加健壮，并与人类的偏好和价值观保持一致。传统上，机器人的行为和目标被训练成包括一组手工制作的特征(即，数据中表示的变量)，这些特征反映了环境中与任务相关的方面。使用精心选择的功能是非常有效的数据效率，但对于人类工程师来说，提前为所有可能重要的功能识别和编写代码是不现实的。从大量数据中训练现代高容量模型是一个很好的选择，只要我们不探索新的(分布外)输入的学习模型。这些模型之所以不能推广到分布外的输入，是因为它们通常无法学习正确的表示，包括重要的特征，而是拾取数据中的虚假模式。这个项目的中心目标是使机器人能够获得目标(因此，行为)的基本正确表示。由于学习目标函数-人类用户想要的-从根本上讲是关于人类的，这项工作提出，只有人类才能确定什么是真正重要的，什么是虚假的。这项研究将引入将机器人的表示与人类对齐的问题。该项目背后的关键观察是，用于学习的传统输入，如演示或比较，旨在教机器人完成全部任务，对于调整机器人的表示并不理想。随着表征对齐被定义为一个问题，就有机会设计新类型的人类反馈，帮助机器人明确地分离正确的表征。该项目将开发新类型的人类反馈和算法，以便有效地向它们学习，以达到一致的表示。前期工作利用这一观察引入了特征轨迹-一种新型的人类输入，用户可以通过它向机器人传授他们关心的特定特征。该项目将追求四个目标，共同解决将机器人表示与人类对齐的各个方面：(1)超越特征痕迹，一次教授一个特征：它将调查将机器人表示与用户对齐的新输入类型，贡献主动学习算法，帮助人类教师提供最有信息量的输入，并建立透明工具，使机器人能够向用户传回他们对表示的当前理解。(2)从新的、特定于表征的人类输入中一次提取所有特征：它将结合自我监督表征学习方法和以人为中心的表征学习方法，研究一次教授完整表征的新人类输入类型。(3)以正确的方式使用正确的表征：给机器人一项新任务，机器人需要学习哪些特征重要，在哪些背景下重要。(4)将早期的工作扩展到政策学习：它将把新的工具扩展到政策学习环境中，并使用以人为本的表征的镜头，以使更好的政策概括适用于新用户，并改善强化学习中的目标错误概括。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project seeks to make robots more robust and aligned with human preferences and values. Traditionally, robot behaviors and objectives were trained to include a set of hand-crafted features (i.e., variables represented in the data) that reflect task-relevant aspects of the environment. Using well-chosen features is very data-efficient, but it is unrealistic for human engineers to identify and write code ahead of time for all the features that could matter. Training modern high-capacity models from a lot of data is a great alternative, as long as we do not probe the learned models on novel (out-of-distribution) inputs. The reason these models fail to generalize to out-of-distribution inputs is that they will generally fail to learn the correct representation, comprising the features that matter, and instead pick up on spurious patterns in the data. The central goal of this project is to enable robots to arrive at the underlying correct representation for objectives (and, hence, behaviors). And since learning the objective function---what the human user wants---is fundamentally about humans, this work proposes that only the human can determine what actually matters vs. what is spurious. The research will introduce the problem of aligning robot representations to humans. The key observation behind the project is that traditional input used in learning, such as demonstrations or comparisons, which is designed to teach the robot the full task, is not ideal for aligning the robot’s representation. With representation alignment defined as a problem, there is the opportunity to design new types of human feedback that help the robot explicitly isolate the right representation. The project will develop new types of human feedback and algorithms for efficiently learning from them to arrive at an aligned representation. Preliminary work leveraged this observation to introduce feature traces---a novel type of human input through which users can teach the robot about specific features they care about. The project will pursue four objectives that together tackle the aspects of aligning robot representations with humans: (1) Teaching one feature at a time, beyond feature traces: It will investigate new input types for aligning robot representations with users, contribute active learning algorithms that help the human teacher provide the most informative input, and build transparency tools that enable robots to teach back to the user their current understanding of the representation. (2) Extracting features all at once from new, representation-specific human input: It will investigate new human input types that teach the full representation all at once by combining self-supervised representation learning methods with human-centric representation learning. (3) Using a correct representation in the right way: Given a new task, the robot needs to learn which features matter and in which contexts. (4) Extending earlier work to policy learning: It will extend new tools to the policy learning setting and use the lens of human-aligned representations to enable better policy generalization to new users and to improve goal mis-generalization in reinforcement learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.