Improved analysis of policy gradient methods in reinforcement learning.

点击翻译按钮获取中文摘要

Reinforcement learning is a popular branch of machine learning that aims to solve a sequential decision-making problem in an environment. This has a wide variety of applications including autonomous driving, robotics, recommendation systems and healthcare. In some of these applications, the cost of a wrong decision could be dramatic. In particular for applications like autonomous driving, the lives of human beings are at stake. As such, it is of crucial importance that we understand how the methods work and whether they really do work in the way that was intended.However, the methods that are used in practice are often only poorly understood. The theory describing these methods is currently unable to explain the huge successes that reinforcement learning has enjoyed in practice. The aim of this project is to provide improved theoretical guarantees for methods known as policy gradient methods that form the basis for much of the practical implementations of reinforcement learning. These methods are particularly used for large-scale problems that are often faced in practice.Specifically, theory on algorithms of this type takes the form of convergence bounds. That is, the algorithm is aiming to output a solution to the problem that is optimal. We are interested in understanding how quickly the algorithm outputs something close to this optimal solution, where the notion of closeness is mathematically precise. The aim of improved analyses translates here into saying that an algorithm converges faster than what was previously proven.Recently, a particular type of a policy-gradient method in a specific setting has been studied under a new perspective known as policy mirror descent. What exactly this means is not too important except that mirror descent is a concept from optimisation theory that has been heavily studied in that setting. As such, tools and methods of analysis may be translated from optimisation theory to this reinforcement learning framework. This can be exploited to achieve improved convergence guarantees, which is one of the avenues that we are using in this project.This project is part of the StatML CDT, which is a joint CDT between Imperial College London and the university of Oxford. It falls within the EPSRC statistics and applied probability research area. In particular, though this project is heavily linked to optimisation, it remains very statistical in nature. This is because we are interested in using data that inherently has some randomness to it in order to solve the decision-making problem of reinforcement learning.