Statistical Methods in Offline Reinforcement Learning

强化学习(RL)关注智能代理如何在给定的环境中采取行动，学习最优策略，使他们获得的累积奖励最大化。在过去的几年里，它可以说是机器学习中最活跃的研究前沿之一。根据谷歌学者的数据，2020年已经发表了超过4万篇科学文章，其中包括短语“强化学习”。超过100篇关于RL的论文被接受在ICML 2020(机器学习领域的首屈一指的会议)上发表，占接受论文总数的10%以上。在使用RL解决各个领域的挑战性问题方面取得了重大进展，包括游戏、机器人、医疗保健、投标和自动驾驶。然而，统计学作为一个领域，而不是计算机科学，只是最近才开始在深度和广度上与RL接触。拟议的研究将开发统计学习方法，以解决离线RL领域的几个关键问题。我们的目标是提出利用以前收集的数据的RL算法，而不需要额外的在线数据收集。这项拟议的研究主要是受到医疗保健应用的推动。现有的大多数最先进的RL算法都是由在线设置(例如，视频游戏)驱动的。它们对医疗保健应用的概括仍不得而知。我们还指出，我们的解决方案将可转移到其他领域(例如，机器人)。拟议中的研究将考虑的一个基本问题是离线政策优化，其目标是学习最优策略，以基于离线数据集最大化长期结果。解决这个问题至少面临两大挑战。首先，与数据易于收集或模拟的在线设置相比，许多离线应用程序(例如医疗保健)中的观察次数是有限的。在数据如此有限的情况下，开发统计上有效的RL算法至关重要。建议的研究将设计出一些普遍适用于最先进的RL算法的“价值增强”方法，以提高其统计效率。对于已有算法计算出的给定初始策略，我们的目标是输出一种新策略，其预期收益以更快的速度收敛，从而实现期望的价值增强特性。其次，许多离线数据集是通过聚合多个异类数据源创建的。在医疗保健中，从不同患者收集的数据轨迹可能不具有共同的分布函数，这是典型的情况。我们将研究RL中现有的迁移学习方法，并基于我们在统计学方面的专业知识，开发适用于医疗保健应用的新方法。另一个拟议的研究将考虑的问题是非政策评估(OPE)。OPE旨在通过预先收集的由不同策略生成的数据集来了解目标策略的预期回报(值)。它在医疗保健和自动驾驶等应用程序中至关重要，在这些应用程序中，需要在在线验证之前离线评估新策略。在现有的大多数著作中，一个常见的假设是没有不可测量的混杂。然而，这一假设并不能从数据中得到检验。它可能会在从医疗保健应用程序生成的观测数据集中被违反。此外，由于样本大小有限，许多离线应用程序将受益于具有量化价值估计器不确定性的置信度区间(CI)。建议的研究关注于在潜在混杂因素存在的情况下为目标政策的价值构建CI。此外，在各种应用中，结果分布是偏态和重尾的。像分位数这样的标准比平均值更合理。我们将开发方法来学习目标政策下回报的分位数曲线，并构建其相关的置信度范围。

Reinforcement learning (RL) is concerned with how intelligent agents take actions in a given environment to learn an optimal policy that maximises the cumulative reward that they receive. It has been arguably one of the most vibrant research frontiers in machine learning over the last few years. According to Google Scholar, over 40K scientific articles have been published in 2020 with the phrase "reinforcement learning". Over 100 papers on RL were accepted for presentation at ICML 2020 (a premier conference in the machine learning area), accounting for more than 10% of the accepted papers in total. Significant progress has been made in solving challenging problems across various domains using RL, including games, robotics, healthcare, bidding and automated driving. Nevertheless statistics as a field, as opposed to computer science, has only recently begun to engage with RL both in depth and in breadth. The proposed research will develop statistical learning methodologies to address several key issues in offline RL domains. Our objective is to propose RL algorithms that utilise previously collected data, without additional online data collection. The proposed research is primarily motivated by applications in healthcare. Most of the existing state-of-the-art RL algorithms were motivated by online settings (e.g., video games). Their generalisations to applications in healthcare remain unknown. We also remark that our solutions will be transferable to other fields (e.g., robotics). A fundamental question the proposed research will consider is offline policy optimisation where the objective is to learn an optimal policy to maximise the long-term outcome based on an offline dataset. Solving this question faces at least two major challenges. First, in contrast to online settings where data are easy to collect or simulate, the number of observations in many offline applications (e.g., healthcare) is limited. With such limited data, it is critical to develop RL algorithms that are statistically efficient. The proposed research will devise some "value enhancement" methods that are generally applicable to state-of-the-art RL algorithms to improve their statistical efficiency. For a given initial policy computed by existing algorithms, we aim to output a new policy whose expected return converges at a faster rate, achieving the desired "value enhancement" property. Second, many offline datasets are created via aggregating over many heterogeneous data sources. This is typically the case in healthcare where the data trajectories collected from different patients might not have a common distribution function. We will study existing transfer learning methods in RL and develop new approaches designed for healthcare applications, based on our expertise in statistics.Another question the proposed research will consider is off-policy evaluation (OPE). OPE aims to learn a target policy's expected return (value) with a pre-collected dataset generated by a different policy. It is critical in applications from healthcare and automated driving where new policies need to be evaluated offline before online validation. A common assumption made in most of the existing works is that of no unmeasured confounding. However, this assumption is not testable from the data. It can be violated in observational datasets generated from healthcare applications. Moreover, many offline applications will benefit from having a confidence interval (CI) that quantifies the uncertainty of the value estimator, due to the limited sample size. The proposed research is concerned with constructing a CI for a target policy's value in the presence of latent confounders. In addition, in a variety of applications, the outcome distribution is skewed and heavy-tailed. Criteria such as quantiles are more sensible than the mean. We will develop methodologies to learn the quantile curve of the return under a target policy and construct its associated confidence band.