Theory and Algorithms for Relation between Stochastic Control and Reinforcement Learning

强化学习是机器学习中一个非常活跃的子领域。近年来，它在复杂决策问题领域取得的现实世界的著名胜利包括玩围棋和驾驶自动驾驶汽车等完美信息棋盘游戏。其基本思想是让算法逐渐学习近乎最佳的策略来实时解决问题。学习是基于改进控制策略的试错法。它要求在黑箱环境下指定一个明确的目标得分函数，该函数结合了环境对控制行动的响应。然而，在许多情况下，这种反复尝试的探索代价高得令人望而却步。RL中的勘探-开采二分法通常通过添加需要作为目标函数的一部分最小化的熵正则化项来缓解。最近建立了熵正则化RL问题与松弛连续时间随机控制之间的联系。特别地，线性二次(LQ)控制问题具有优雅的解，并且能够逼近更一般的问题。其中一个目标是设计和实现求解该问题的算法，无论是LQ问题还是一般控制问题，并给出相应的收敛定理。为此，Szpruch等人提出了算法。和Reisinger等人。都是很好的例子，可以作为一个起点。在实际问题中，带跳的随机过程有着广泛的应用。因此，这个项目的另一个目标是考虑更复杂的受控动态，即带有跳跃的过程，并看看我们的算法是否能够解决这个更复杂的问题。Hernandez-Hernandez等人提出的鞅方法。可以应用于这个项目。在本研究中，我们将主要对这一问题进行理论研究。当理论比较完善时，我们将尝试开发相应的算法，并给出它们的收敛定理。有了这些理论和算法，我们将继续这个项目的最后一个目标：将这些算法应用于不同的现实世界问题，如金融交易、自动驾驶和棋类游戏。RL在金融中的许多应用还没有被纳入连续时间松弛随机控制框架。预计在本博士期间，将把理论上的进展应用于量化金融中的问题(例如，投资组合交易的最优执行和衍生证券的风险管理)，重点放在算法的现实适用性上。

Reinforcement learning (RL) is an extremely active subfield in machine learning. Its famous real-world triumphs in the area of complex decision making problems in recent years include playing perfect-information board games such as Go and driving autonomous vehicles. The basic idea is for an algorithm to learn gradually near-best strategies to solve a problem in real time. The learning is based on trial-and-error for improving control policies. It requires specifying an explicit objective score function on a black-box environment, which incorporates the environmental responses to the control actions. However, such trial-and-error exploration is in many situations prohibitively costly. The exploration-exploitation dichotomy in RL is typically mitigated by adding an entropy-regularisation term that needs to be minimised as part of the objective function. The connection between entropy-regularized RL problems and the relaxed continuous-time stochastic control has been established recently. In particular, Linear-Quadratic (LQ) control problem has elegant solutions and is able to approximate more general problems. One of the objective is to design and implement algorithms for solving this problem, either LQ problem or general control problem, and provide corresponding convergence theorems. To this end, algorithms proposed by Szpruch et al. and Reisinger et al. are good examples and could be a starting point. In real-world problems, stochastic process with jumps is widely applied. Thus, another objective of this project is to consider more complex controlled dynamics, i.e. processes with jump, and see whether our algorithms could tackle this more complex problem. The martingale approach proposed by Hernandez-Hernandez et al. could be applied to this project. In this research, we will mainly study the theory of the problem. When the theory is relatively complete, we will try to develop corresponding algorithms and provide convergence theorems for them. With these theory and algorithms, we will proceed to the last objective of this project: applying the algorithms to different real-world problem, such as financial trading, autonomous driving and board games. Many applications of RL in finance are yet to be incorporated in the continuous-time relaxed stochastic control framework. It is envisaged that, during this PhD, theoretical advances will be applied to problems in quantitative finance (e.g. optimalexecution of portfolio transactions and the risk-management of derivative securities) with emphasis on real-worlds applicability of the algorithms.