Scalable Autonomous Reinforcement Learning - From scratch to less and less structure

在过去的十年中，强化学习（RL）的框架已经发展成为一种很有前途的工具，用于学习机器人中各种不同的任务。在此期间，强化学习在扩展到高维系统和解决日益复杂的任务方面取得了很大进展。不幸的是，这种可扩展性已经通过使用专家知识在多个维度上预先构建学习问题来实现。因此，机器人强化学习中最先进的方法通常依赖于手工制作的状态表示，预先结构化的参数化策略，良好形状的奖励函数和人类专家的演示，以帮助扩展学习算法。在这个提议中，我们希望通过从一个具有挑战性的机器人任务的“经典”强化学习设置开始（即，tetherball）。通过RL方法解决这个任务已经是一个有价值的贡献。从那里开始，我们将开始确定学习任务设计仍然需要工程经验的组件。在这个提议的过程中，我们展示了我们的目标是如何驱动这些组件中的每一个走向更多的自治，同时开发高度可扩展的方法。为此，我们将开发系统的方法来增加学习系统的自治性，超越传统的方法：（1）提出自动学习强化学习状态表示的方法;（2）开发能够表示真正自主行为所必需的大量控制策略的通用策略类;（3）自主地发现信息奖励函数。这些方面的进步将把学习算法提升到更高的自主水平。这些进展将基于完善的政策搜索理论框架，并通过改进最先进的强化学习算法来实现。最终，生成的系统应该学会如何从简单、通用的原则将原始感觉输入映射到原始控制信号，自动发现其环境中的结构，并在没有专业知识的情况下解决困难的控制任务。如果成功的话，该项目中开发的完整方法及其子部分将有助于建立新一代更强大的强化学习算法，能够自主解决复杂的机器人控制问题。

Over the course of the last decade, the framework of reinforcement learning (RL) has developed into a promising tool for learning a large variety of different tasks in robotics. During this timeframe, a lot of progress has been made towards scaling reinforcement learning to high-dimensional systems and solving tasks of increasing complexity. Unfortunately, this scalability has been achieved by using expert knowledge to pre-structure the learning problem in several dimensions. As a consequence, the state-of-the-art methods in robot reinforcement learning generally depend on hand-crafted state representations, pre-structured parametrized policies, well-shaped reward functions and demonstrations by a human expert to aid scaling of the learning algorithm.In this proposal, we want to advance the field by starting with a 'classical' reinforcement learning setting for a challenging robotic task (i.e., tetherball). Solving this task by RL methods will be already a valuable contribution. From there on, we will start to identify the components for which the learning task design still needs engineering experience. In the course of this proposal, we show how we aim to drive each of these components towards more autonomy while developing highly scalable approaches.To this end, we will develop systematic methods to increase the autonomy of the learning system by going beyond traditional approaches: (1) proposing methods for learning state representations for reinforcement learning automatically; (2) developing generic policy classes capable of representing the large variety of control policies that are necessary for truly autonomous behavior; (3) discovering informative reward functions autonomously. Progress in each of these aspects will lift the learning algorithm to a higher level of autonomy. The advances will be grounded in the well established theoretical framework of policy search and enabled through improvements to state-of-the-art reinforcement learning algorithms. Ultimately the resulting system should learn how to map raw sensory inputs to raw control signals from simple, generic principles, discovering structure within its environment automatically and solving difficult control tasks without expert knowledge. If successful, both the complete methodology developed within this project as well as sub-parts of it will help to establish a new, substantially more powerful generation of reinforcement learning algorithms that are capable of solving complicated robot control problems autonomously.