Collaborative Research: HCC: Medium: Aligning Robot Representations with Humans

该项目旨在使机器人更加健壮，并与人类的偏好和价值观保持一致。传统上，机器人的行为和目标被训练成包括一组手工制作的特征（即数据中表示的变量），这些特征反映了环境中与任务相关的方面。使用精心选择的功能是非常有效的数据，但对于人类工程师来说，提前为所有可能重要的功能识别和编写代码是不现实的。从大量数据中训练现代高容量模型是一个很好的选择，只要我们不在新的（非分布的）输入上探测学习到的模型。这些模型无法推广到分布外输入的原因是，它们通常无法学习正确的表示，包括重要的特征，而是在数据中挑选虚假的模式。该项目的中心目标是使机器人能够获得目标（以及行为）的底层正确表示。由于学习目标函数——人类用户想要什么——从根本上讲是关于人类的，这项工作提出，只有人类才能确定什么是真正重要的，什么是虚假的。该研究将引入将机器人表征与人类表征对齐的问题。该项目背后的关键观察是，用于学习的传统输入，如演示或比较，旨在教会机器人完成完整的任务，对于对齐机器人的表示并不理想。将表示对齐定义为一个问题，就有机会设计新型的人类反馈，帮助机器人明确地隔离正确的表示。该项目将开发新型的人类反馈和算法，以便有效地从中学习，从而获得一致的表示。初步工作利用这一观察引入了特征跟踪——一种新型的人类输入，用户可以通过这种输入告诉机器人他们关心的特定特征。该项目将追求四个目标，共同解决机器人表征与人类表征的对齐方面：(1)一次教授一个特征，超越特征轨迹：它将研究用于将机器人表征与用户对齐的新输入类型，贡献主动学习算法，帮助人类教师提供最具信息量的输入，并构建透明工具，使机器人能够向用户传授他们当前对表征的理解。(2)从新的、特定于表示的人类输入中一次性提取特征：它将研究新的人类输入类型，通过将自我监督的表示学习方法与以人为中心的表示学习相结合，一次性教授完整的表示。(3)以正确的方式使用正确的表征：给定一个新任务，机器人需要学习哪些特征是重要的，在哪些环境中。(4)将早期的工作扩展到策略学习：它将扩展新的工具到策略学习设置，并使用与人类一致的表示的镜头来实现对新用户的更好的策略泛化，并改善强化学习中的目标错误泛化。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project seeks to make robots more robust and aligned with human preferences and values. Traditionally, robot behaviors and objectives were trained to include a set of hand-crafted features (i.e., variables represented in the data) that reflect task-relevant aspects of the environment. Using well-chosen features is very data-efficient, but it is unrealistic for human engineers to identify and write code ahead of time for all the features that could matter. Training modern high-capacity models from a lot of data is a great alternative, as long as we do not probe the learned models on novel (out-of-distribution) inputs. The reason these models fail to generalize to out-of-distribution inputs is that they will generally fail to learn the correct representation, comprising the features that matter, and instead pick up on spurious patterns in the data. The central goal of this project is to enable robots to arrive at the underlying correct representation for objectives (and, hence, behaviors). And since learning the objective function---what the human user wants---is fundamentally about humans, this work proposes that only the human can determine what actually matters vs. what is spurious. The research will introduce the problem of aligning robot representations to humans. The key observation behind the project is that traditional input used in learning, such as demonstrations or comparisons, which is designed to teach the robot the full task, is not ideal for aligning the robot’s representation. With representation alignment defined as a problem, there is the opportunity to design new types of human feedback that help the robot explicitly isolate the right representation. The project will develop new types of human feedback and algorithms for efficiently learning from them to arrive at an aligned representation. Preliminary work leveraged this observation to introduce feature traces---a novel type of human input through which users can teach the robot about specific features they care about. The project will pursue four objectives that together tackle the aspects of aligning robot representations with humans: (1) Teaching one feature at a time, beyond feature traces: It will investigate new input types for aligning robot representations with users, contribute active learning algorithms that help the human teacher provide the most informative input, and build transparency tools that enable robots to teach back to the user their current understanding of the representation. (2) Extracting features all at once from new, representation-specific human input: It will investigate new human input types that teach the full representation all at once by combining self-supervised representation learning methods with human-centric representation learning. (3) Using a correct representation in the right way: Given a new task, the robot needs to learn which features matter and in which contexts. (4) Extending earlier work to policy learning: It will extend new tools to the policy learning setting and use the lens of human-aligned representations to enable better policy generalization to new users and to improve goal mis-generalization in reinforcement learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.