Collaborative Research: RI: Medium: Superhuman Imitation Learning from Heterogeneous Demonstrations

从表现出的行为中学习（即，模仿）是动物和人类知识转移的有效手段。现有的人工智能（AI）系统的模仿学习方法通常假设模仿者的能力与演示者的能力相匹配。当模仿者的能力显着超过演示者的能力时，这可能会导致不良行为。该项目重新制定了在某些方面比（人类）演示者更有能力的人工智能系统的模仿学习，试图使人工智能系统明确优于人类演示者。该项目将培训研究生和本科生开发人工智能系统，使其更好地符合广泛的高度影响力的未来应用中的安全和实用要求。该项目使用最大裕度优化来指导控制/决策策略的（深度）强化学习，从而实现其重新制定的模仿学习目标。它侧重于从质量、难度和结构不同的异构演示和任务中学习。最初，用于评估和比较不同行为的多个度量被假定为可用。在项目的后期，这些指标将使用深度表示学习方法从演示和补充注释中学习。该项目的方法产生的政策将在一组不同的应用程序上进行评估：开源模拟器（例如，该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Learning from demonstrated behavior (i.e., imitation) is an effective means of knowledge transfer in animals and humans. Existing imitation learning methods for artificial intelligence (AI) systems typically assume the capabilities of the imitator match those of the demonstrator. This can lead to undesirable behavior when the imitator’s capabilities significantly exceed those of the demonstrator. This project reformulates imitation learning for AI systems that are more capable than (human) demonstrators in some aspects by seeking to make the AI system unambiguously better than human demonstrators. The project will train graduate students and undergraduates to develop artificial intelligence systems that are better aligned with safety and utility requirements in a broad range of highly impactful future applications.The project approaches its reformulated imitation learning objective using a maximum margin optimization for guiding (deep) reinforcement learning of control/decision policies. It focuses on learning from heterogeneous demonstrations and tasks that differ in quality, difficulty, and structure. Initially, multiple metrics for assessing and comparing different behaviors are assumed to be available. Later in the project, these metrics will be learned from demonstrations and supplemental annotations using deep representation learning methods. The policies produced by the approach of this project will be evaluated on a diverse set of applications: open source simulators (e.g., Atari games), manipulation and mobility tasks for robotics platforms, and cancer treatment decisions.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.