RI: Small: Feature Encoding for Reinforcement Learning

RI：小型：强化学习的特征编码

• 批准号: 1815300
• 负责人: Ronald Parr
• 金额: $ 50万
• 依托单位: Duke University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-08-01 至 2023-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1815300&HistoricalAwards=false
• 关键词: RI; Small; Feature; Encoding; Reinforcement

This project focuses on the subfield of machine learning referred to as Reinforcement Learning (RL), in which algorithms or robots learn by trial and error. As with many areas of machine learning, there has been a surge of interest in "deep learning" approaches to reinforcement learning, i.e, "Deep RL." Deep learning uses computational models motivated by structures found in the brains of animals. Deep RL has enjoyed some stunning successes, including a recent advance by which a program learned to play the Asian game of Go better than the best human player. Notably, this level of performance was achieved without any human guidance. Given only the rules of the game, the program learned by playing against itself. Although games are intriguing and attention-grabbing, this feat was merely a technology demonstration. Firms are seeking to deploy Deep RL methods to increase the efficiency of their operations across a range of applications such as data center management and robotics. To realize fully the potential of Deep RL, further research is required to make the training process more predictable, reliable, and efficient. Current techniques require massive amounts of training data and computation, and subtle changes in the configuration of the system can cause huge differences in the quality of the results obtained. Thus, even though RL systems can learn autonomously by trial and error, a large amount of human intuition, experience and experimentation may be required to lay the groundwork for these systems to succeed. This proposal seeks to develop new techniques and theory to make high quality deep RL results more widely and easily obtainable. In addition, this proposal will provide opportunities for undergraduates to be involved in research through Duke's Data+ initiative.The proposed research is partly inspired by past work on feature selection and discovery for reinforcement learning. Much of that work focused primarily on linear value function approximation. Its relevance to deep reinforcement learning is that methods such as Deep Q-learning have a linear final layer. The preceding, nonlinear layers can, therefore, be interpreted as performing feature discovery for what is ultimately a linear value function approximation process. Sufficient conditions on the features that were specified for successful linear value function approximation in earlier work can now be re-interpreted as an intermediate objective function for the penultimate layer of a deep network. The proposed research aims to achieve the following objectives: 1) Develop a theory of feature construction that explains and informs deep reinforcement learning methods, 2) develop improved approaches to value function approximation that are applicable to deep reinforcement learning, 3) develop improved approaches to policy search that are applicable to deep reinforcement learning, and 4) develop new algorithms for exploration in reinforcement learning that take advantage of learned feature representations, and 5) perform computational experiments demonstrating the efficacy of the new algorithms developed on benchmark problems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目重点关注机器学习的子领域，称为强化学习（RL），其中算法或机器人通过反复试验来学习。与机器学习的许多领域一样，人们对强化学习的“深度学习”方法（即“深度强化学习”）兴趣激增。 深度学习使用由动物大脑中发现的结构驱动的计算模型。深度强化学习取得了一些惊人的成功，包括最近的一项进展，一个程序学会了比最好的人类棋手更好地玩亚洲围棋游戏。值得注意的是，这种性能水平是在没有任何人工指导的情况下实现的。仅给定游戏规则，程序就通过与自身对弈来学习。尽管游戏很有趣且引人注目，但这一壮举只是一次技术演示。公司正在寻求部署深度强化学习方法，以提高数据中心管理和机器人等一系列应用的运营效率。为了充分发挥深度强化学习的潜力，需要进一步研究以使训练过程更加可预测、可靠和高效。当前的技术需要大量的训练数据和计算，系统配置的细微变化可能会导致所获得结果的质量存在巨大差异。因此，尽管强化学习系统可以通过反复试验自主学习，但可能需要大量的人类直觉、经验和实验来为这些系统的成功奠定基础。该提案旨在开发新技术和理论，使高质量的深度强化学习结果更广泛、更容易获得。此外，该提案将为本科生提供通过杜克大学数据+计划参与研究的机会。拟议研究的部分灵感来自于过去关于强化学习的特征选择和发现的工作。 大部分工作主要集中在线性值函数近似上。 它与深度强化学习的相关性在于深度 Q 学习等方法具有线性最终层。 因此，前面的非线性层可以解释为执行特征发现，以实现最终的线性值函数近似过程。 在早期工作中为成功的线性值函数近似指定的特征的充分条件现在可以重新解释为深度网络倒数第二层的中间目标函数。拟议的研究旨在实现以下目标：1）开发一种特征构建理论，解释和告知深度强化学习方法，2）开发适用于深度强化学习的价值函数逼近的改进方法，3）开发适用于深度强化学习的改进的策略搜索方法，4）开发利用学习到的特征表示的强化学习探索新算法，5）进行计算实验证明 该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Policy Caches with Successor Features

• 发表时间: 2021
• 作者: Mark W. Nemecek;R. Parr