RI: Small: Combining Reinforcement Learning and Deep Learning Methods to Address High-Dimensional Perception, Partial Observability and Delayed Reward

考虑一个机器代理所面临的问题，它必须与一些动态环境进行交互以实现某些目标。具体地说，想象一个智能体像人类一样参与虚拟竞争。它可以看到由许多移动物体组成的屏幕。在任何时候，它都可以从十几个动作中选择一个。它的动作控制屏幕上的一个对象，但通常不清楚是哪一个。每隔一段时间就会对比赛进行一次评价.在某种程度上，竞争结束了。这样的代理应该如何选择动作，或者更重要的是，我们如何构建可以学习竞争的代理，即，通过反复试验获得高分。在这个项目中，将开发和评估方法，以建立这样的代理。 上述问题是所谓的强化学习（RL）问题的一个实例。这样的问题在顺序决策环境中比比皆是。在工业中的应用包括工厂优化、机器人和慢性疾病管理（仅列出三个不同的感兴趣领域）。像许多这些RL问题一样，Atari游戏（这里用作测试平台来评估学习策略）有三个对本项目感兴趣的特征。首先，它们生成高维图像，因此智能体面临困难的感知问题。第二，他们通常有深度延迟的奖励;即，这些行动会产生长期的后果。例如，失去一种资源可能在损失的那一刻并不需要花费，但在很久以后，当这种资源是非常必要的时候，可能会导致非常高的损失。第三，它们具有深度部分可观测性，即，要有效地竞争，就必须经常记住深刻的过去。例如，在过去很久以前遇到的位置可能在很久以后变得有价值，因为关键资源在那时变得可用，并且代理将必须找到返回该位置的路以使用该资源。建议分别用新的神经网络架构来解决这三个挑战，新的神经网络架构用于预测行动的后果，新的方法用于内在地激励代理，即使奖励被延迟，以及新的递归神经网络架构来有效地记住过去。预计拟议工作的成功将大大扩大强化学习的应用范围。最后，Atari游戏将代替工厂优化作为评估域，因为它们很容易获得。 他们将被用来吸引高中和代表性不足的本科生的兴趣到复杂的想法背后的拟议工作;他们有趣的可视化将允许他们被整合到教学中的PI的类，有各种各样的游戏，在三个挑战维度的难度程度不同，允许更有效地控制评估更有效。

Consider the problem faced by a machine agent that has to interact with some dynamical environment to achieve some goals. Concretely, imagine an agent engaged in a virtual competition as a human would. It can see the screen composed of many moving objects. At any time, it can choose one of a dozen or so actions. Its action controls one of the objects on the screen, but it often is not clear which one. Every so often the an evaluation is given of the competition. At some point the competition ends. How should such an agent choose actions, or more importantly how can we build agents that can learn to compete, i.e., achieve high scores, through trial and error. In this project methods will be developed and evaluated to build such agents. The above problem is an instance of what is called a reinforcement learning (RL) problem. Such problems abound in sequential decision-making settings. Applications in industry include factory optimization, robotics, and chronic disease management (to list but three diverse domains of interest). Like many of these RL problems, Atari games (used as a testbed here to evaluate learning strategies) have three characteristics of interest to this project. First, they generate high-dimensional images and so the agent faces a difficult perception problem. Second, they often have deeply-delayed rewards; i.e., actions have long-term consequences. For example, losing a resource may not cost at the moment of loss, but could lead to very high losses much later when that resource is critically necessary. Third, they have deep partial observability, i.e., to compete effectively one has to often remember the deep past. For example, a location encountered far back in the past may become valuable much later because a critical resource becomes available at that time and the agent would have to find its way back to that location to use the resource. It is proposed to address these three challenges respectively with new neural network architectures for predicting the consequences of actions, new methods for intrinsically motivating agents even when reward is delayed, and new recurrent neural network architectures to remember the past effectively. Success of the proposed work is expected to significantly expand the scope of application of reinforcement learning. Finally, Atari games will be used instead of, say, factory optimization as an evaluation domain because they are readily available. They will be used to draw high-school and under-represented undergraduate students interest into complex ideas underlying the proposed work; their fun visualizations will allow them to be integrated into teaching in the PIs' classes, and there are a variety of games that vary in the degree of difficulty of the three challenge dimensions allowing more effective control of the evaluations more effectively.