RI: Small: Feature Encoding for Reinforcement Learning

本项目专注于机器学习的子领域，称为强化学习(RL)，在该领域中，算法或机器人通过反复试验进行学习。与机器学习的许多领域一样，人们对强化学习的“深度学习”方法也产生了浓厚的兴趣，即“深度RL”。深度学习使用由动物大脑中发现的结构驱动的计算模型。Deep RL已经取得了一些惊人的成功，包括最近的一项进步，一个程序学会了比最好的人类棋手更好地下亚洲围棋。值得注意的是，这种水平的表现是在没有任何人类指导的情况下实现的。在只知道游戏规则的情况下，程序是通过与自己对抗来学习的。尽管游戏引人入胜，吸引眼球，但这一壮举只是一次技术演示。公司正在寻求部署Deep RL方法，以提高数据中心管理和机器人等一系列应用程序的运营效率。为了充分发挥Deep RL的潜力，需要进一步研究，使培训过程更加可预测、可靠和高效。目前的技术需要大量的训练数据和计算，系统配置的细微变化可能会导致所获得的结果质量的巨大差异。因此，尽管RL系统可以通过试错自主学习，但可能需要大量的人类直觉、经验和实验来为这些系统的成功奠定基础。这一建议旨在开发新的技术和理论，使高质量的深度RL结果更广泛和更容易获得。此外，这项建议将通过杜克的数据+倡议为本科生提供参与研究的机会。拟议的研究在一定程度上受到了过去关于强化学习的特征选择和发现的工作的启发。大部分工作主要集中在线性值函数逼近上。它与深度强化学习的相关性在于，深度Q-学习等方法有一个线性的最后一层。因此，前面的非线性层可以被解释为执行最终为线性值函数近似过程的特征发现。在以前的工作中，关于成功逼近线性值函数的特征的充分条件现在可以被重新解释为深层网络倒数第二层的中间目标函数。这项研究旨在实现以下目标：1)发展一种解释和告知深度强化学习方法的特征构建理论；2)开发适用于深度强化学习的改进的值函数逼近方法；3)开发适用于深度强化学习的改进的策略搜索方法；4)开发新的利用学习特征表示的强化学习探索算法；5)进行计算实验，展示基于基准问题开发的新算法的有效性。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project focuses on the subfield of machine learning referred to as Reinforcement Learning (RL), in which algorithms or robots learn by trial and error. As with many areas of machine learning, there has been a surge of interest in "deep learning" approaches to reinforcement learning, i.e, "Deep RL." Deep learning uses computational models motivated by structures found in the brains of animals. Deep RL has enjoyed some stunning successes, including a recent advance by which a program learned to play the Asian game of Go better than the best human player. Notably, this level of performance was achieved without any human guidance. Given only the rules of the game, the program learned by playing against itself. Although games are intriguing and attention-grabbing, this feat was merely a technology demonstration. Firms are seeking to deploy Deep RL methods to increase the efficiency of their operations across a range of applications such as data center management and robotics. To realize fully the potential of Deep RL, further research is required to make the training process more predictable, reliable, and efficient. Current techniques require massive amounts of training data and computation, and subtle changes in the configuration of the system can cause huge differences in the quality of the results obtained. Thus, even though RL systems can learn autonomously by trial and error, a large amount of human intuition, experience and experimentation may be required to lay the groundwork for these systems to succeed. This proposal seeks to develop new techniques and theory to make high quality deep RL results more widely and easily obtainable. In addition, this proposal will provide opportunities for undergraduates to be involved in research through Duke's Data+ initiative.The proposed research is partly inspired by past work on feature selection and discovery for reinforcement learning. Much of that work focused primarily on linear value function approximation. Its relevance to deep reinforcement learning is that methods such as Deep Q-learning have a linear final layer. The preceding, nonlinear layers can, therefore, be interpreted as performing feature discovery for what is ultimately a linear value function approximation process. Sufficient conditions on the features that were specified for successful linear value function approximation in earlier work can now be re-interpreted as an intermediate objective function for the penultimate layer of a deep network. The proposed research aims to achieve the following objectives: 1) Develop a theory of feature construction that explains and informs deep reinforcement learning methods, 2) develop improved approaches to value function approximation that are applicable to deep reinforcement learning, 3) develop improved approaches to policy search that are applicable to deep reinforcement learning, and 4) develop new algorithms for exploration in reinforcement learning that take advantage of learned feature representations, and 5) perform computational experiments demonstrating the efficacy of the new algorithms developed on benchmark problems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.