Collaborative Research: RI: Medium: Bootstrapping natural feedback for reinforcement learning

合作研究：RI：中：引导强化学习的自然反馈

• 批准号: 2212310
• 负责人: Jacob Andreas
• 金额: $ 120万
• 依托单位: Massachusetts Institute of Technology
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-09-01 至 2025-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2212310&HistoricalAwards=false
• 关键词: Collaborative; Research; RI; Medium; Bootstrapping

Many modern applications of artificial intelligence---from industrial automation to content recommendation---depend on machine learning algorithms that train automated agents to interact with their environments. But the two main approaches to interactive learning, reinforcement learning and imitation, require so much supervision or training time that it is prohibitively expensive to apply them to most real-world problems. Human learning does not suffer from this shortcoming, in large part because humans learn not from rewards or demonstrations, but instead from extended interaction with skilled teachers who use signals like gesture and language. This project will lay a foundation for research on interactive learning with rich feedback, from the perspective of individual agents, human--agent teams, and multi-agent populations. It will yield new capabilities for interactive training of automated agents, expanding both the effectiveness and accessibility of such techniques. Support for natural, interactive feedback will also improve the customizability of such systems, making on-the-fly adaptation or retraining accessible to users without significant computing power, data annotation resources or even programming ability.The project is organized into three broad research objectives. First, it will develop a formal framework for grounding feedback, using simple supervisory signals (provided during or after execution) to bootstrap learned interpretation of more complex feedback types. Second, it will develop algorithms for learning to solicit feedback. These algorithms will turn the one-way process of reinforcement learning into a two-way interaction, enabling agents to proactively query supervisors for information about the compositional and causal structure of the environment. Third, it will develop new mechanisms and techniques for providing feedback, via software tools that assist human supervisors in selecting or generating maximally informative feedback signals. Research under each of these objectives will be carried out in simulated environments, benchmarked using complex tasks spanning navigation, robot manipulation, and furniture assembly, and evaluated in terms of its benefits to sample efficiency, end-to-end development time, and usability.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

人工智能的许多现代应用--从工业自动化到内容推荐--都依赖于机器学习算法，这些算法可以训练自动代理与环境进行交互。但是交互式学习的两种主要方法，强化学习和模仿，需要大量的监督或训练时间，以至于将它们应用于大多数现实世界的问题是非常昂贵的。人类的学习并不受这个缺点的影响，这在很大程度上是因为人类不是从奖励或示范中学习，而是通过与熟练的教师进行广泛的互动来学习，这些教师使用手势和语言等信号。该项目将从个体代理、人-代理团队和多代理群体的角度，为具有丰富反馈的交互式学习研究奠定基础。 它将为自动代理的交互式培训提供新的能力，扩大这种技术的有效性和可访问性。支持自然的、交互式的反馈也将提高这种系统的可定制性，使用户在没有强大的计算能力、数据注释资源甚至编程能力的情况下也能进行动态适应或再培训。首先，它将开发一个正式的框架，用于接地反馈，使用简单的监督信号（在执行过程中或执行后提供）引导学习更复杂的反馈类型的解释。其次，它将开发学习征求反馈的算法。这些算法将把强化学习的单向过程变成双向交互，使智能体能够主动向监督者查询有关环境组成和因果结构的信息。第三，它将开发提供反馈的新机制和技术，通过软件工具帮助人类监督者选择或生成最大信息反馈信号。每个目标下的研究将在模拟环境中进行，使用跨越导航，机器人操作和家具装配的复杂任务进行基准测试，并根据其对样品效率，端到端开发时间，该奖项反映了NSF的法定使命，并被认为是值得通过使用基金会的知识价值和更广泛的影响审查评估的支持的搜索.

项目成果

Guiding Pretraining in Reinforcement Learning with Large Language Models

• DOI: 10.48550/arxiv.2302.06692
• 发表时间: 2023-02
• 作者: Yuqing Du;Olivia Watkins;Zihan Wang;Cédric Colas;Trevor Darrell;P. Abbeel;Abhishek Gupta;Jacob Andreas