RI: Small: Feature Encoding for Reinforcement Learning

该项目专注于机器学习的子领域，即强化学习（RL），其中算法或机器人通过试错来学习。与机器学习的许多领域一样，人们对强化学习的“深度学习”方法（即“深度强化学习”）的兴趣激增。深度学习使用由动物大脑结构驱动的计算模型。深度强化学习已经取得了一些惊人的成功，包括最近的一个进步，一个程序学会了比最好的人类棋手更好地下亚洲围棋。值得注意的是，这种水平的表现是在没有任何人为指导的情况下实现的。只要给定游戏规则，程序就会通过与自己对弈来学习。尽管游戏很吸引人，也很吸引眼球，但这一壮举仅仅是一种技术演示。公司正在寻求部署深度强化学习方法，以提高其在数据中心管理和机器人等一系列应用中的运营效率。为了充分发挥深度强化学习的潜力，需要进一步研究使训练过程更加可预测、可靠和高效。目前的技术需要大量的训练数据和计算，系统配置的细微变化会导致所获得的结果质量的巨大差异。因此，尽管强化学习系统可以通过试错自主学习，但可能需要大量的人类直觉、经验和实验来为这些系统的成功奠定基础。本提案旨在发展新的技术和理论，使高质量的深度强化学习结果更广泛和更容易获得。此外，该提案将通过杜克大学的Data+计划为本科生提供参与研究的机会。提出的研究部分受到了过去关于强化学习的特征选择和发现的工作的启发。大部分工作主要集中在线性值函数近似上。它与深度强化学习的相关性在于，像深度q学习这样的方法有一个线性的最终层。因此，前面的非线性层可以被解释为执行最终是线性值函数近似过程的特征发现。在早期的工作中，为成功的线性值函数近似指定的特征的充分条件现在可以重新解释为深度网络倒数第二层的中间目标函数。建议的研究旨在达到以下目标：1)发展特征构建理论，解释和指导深度强化学习方法；2)发展适用于深度强化学习的改进的值函数逼近方法；3)发展适用于深度强化学习的改进的策略搜索方法；4)开发用于强化学习探索的新算法，利用学习到的特征表示。5)进行计算实验，证明新算法在基准问题上的有效性。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project focuses on the subfield of machine learning referred to as Reinforcement Learning (RL), in which algorithms or robots learn by trial and error. As with many areas of machine learning, there has been a surge of interest in "deep learning" approaches to reinforcement learning, i.e, "Deep RL." Deep learning uses computational models motivated by structures found in the brains of animals. Deep RL has enjoyed some stunning successes, including a recent advance by which a program learned to play the Asian game of Go better than the best human player. Notably, this level of performance was achieved without any human guidance. Given only the rules of the game, the program learned by playing against itself. Although games are intriguing and attention-grabbing, this feat was merely a technology demonstration. Firms are seeking to deploy Deep RL methods to increase the efficiency of their operations across a range of applications such as data center management and robotics. To realize fully the potential of Deep RL, further research is required to make the training process more predictable, reliable, and efficient. Current techniques require massive amounts of training data and computation, and subtle changes in the configuration of the system can cause huge differences in the quality of the results obtained. Thus, even though RL systems can learn autonomously by trial and error, a large amount of human intuition, experience and experimentation may be required to lay the groundwork for these systems to succeed. This proposal seeks to develop new techniques and theory to make high quality deep RL results more widely and easily obtainable. In addition, this proposal will provide opportunities for undergraduates to be involved in research through Duke's Data+ initiative.The proposed research is partly inspired by past work on feature selection and discovery for reinforcement learning. Much of that work focused primarily on linear value function approximation. Its relevance to deep reinforcement learning is that methods such as Deep Q-learning have a linear final layer. The preceding, nonlinear layers can, therefore, be interpreted as performing feature discovery for what is ultimately a linear value function approximation process. Sufficient conditions on the features that were specified for successful linear value function approximation in earlier work can now be re-interpreted as an intermediate objective function for the penultimate layer of a deep network. The proposed research aims to achieve the following objectives: 1) Develop a theory of feature construction that explains and informs deep reinforcement learning methods, 2) develop improved approaches to value function approximation that are applicable to deep reinforcement learning, 3) develop improved approaches to policy search that are applicable to deep reinforcement learning, and 4) develop new algorithms for exploration in reinforcement learning that take advantage of learned feature representations, and 5) perform computational experiments demonstrating the efficacy of the new algorithms developed on benchmark problems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.