Learning good representations for and with reinforcement learning

人工智能(AI)在分离智能的不同方面取得了很大进展，提出了灵活的表示和强大的算法，从而在特定任务中具有能力。例如，人工智能特工比人类更擅长下围棋这样的游戏，这曾被认为是不可能的壮举。然而，人类甚至动物惯常表现出的那种灵活、健壮和自主的能力仍然难以捉摸。最好的人工智能系统仍然针对特定的问题进行调整。我们的主要研究目标是发展通用的人工智能方法论，其核心依赖于强化学习。强化学习是在动物学习理论的启发下，从与环境的交互中学习的一种方法。该建议旨在设计能够自动创建强化学习代理的表示的算法，从而允许它们对世界进行建模并在多个时间尺度上行动。我们的目标是提供新的优化标准，形式化地描述什么是一组好的抽象表示，提供基于梯度的学习算法来学习这样的模型，并通过在模拟领域、博弈以及实时序列预测数据集上的经验评估来证明它们的有效性。我们将解决探索的关键问题，解释代理应该如何在其环境中移动，以优化其学习速度。最后，我们将在其他算法中利用这些方法，这些算法可以受益于多个时间尺度，例如深层递归神经网络的训练。

Artificial intelligence (AI) has made great progress in isolating different aspects of intelligence and proposing flexible representations and powerful algorithms that lead to competence in specific tasks. For example, AI agents are better than humans at playing games like Go, a feat once considered impossible. However, the sort of flexible, robust, and autonomous competence routinely exhibited by humans, or even animals, remains elusive. The best AI systems are still tuned to specific problems. Our main research goal is to develop general AI methodology that relies, at its core, on reinforcement learning. Reinforcement learning is an approach to learning from interaction with an environment, inspired by animal learning theory. This proposal aims to design algorithms that can automatically create representations for reinforcement learning agents which allow them to model the world and to act at multiple time scales. We aim to provide new optimization criteria which describe formally what is a good set of abstract representations, provide gradient-based learning algorithms to learn such models, and demonstrate their effectiveness through empirical evaluations in simulated domains, game playing, as well as real time series prediction data sets. We will tackle the crucial problem of exploration, by explaining how an agent should move about its environment in order to optimize its learning speed. Finally, we will leverage these methods inside other algorithms that can benefit from multiple time scales, such as the training of deep, recurrent neural networks.