Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizons
Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizons
复制标题
DOI：
10.1080/01621459.2022.2106868
复制
发表时间：
2022-02
影响因子：
3.7
通讯作者：
C. Shi;S. Luo;Hongtu Zhu;R. Song
中科院分区：
文献类型：
--
作者：
C. Shi;S. Luo;Hongtu Zhu;R. Song
摘要我们考虑将强化学习(RL)方法应用于离线领域，而不需要额外的在线数据收集，例如移动健康应用。计算机科学文献中的大多数现有策略优化算法都是在数据易于收集或模拟的在线环境中开发的。他们对移动健康应用程序的概括仍然是预先收集的离线数据集，但探索得较少。本文的目的是开发一种新的优势学习框架，以便有效地利用预先收集的数据进行政策优化。该方法以任何现有RL算法计算出的最优Q估计器作为输入，输出一个新的策略，其值保证比基于初始Q估计器得到的策略更快地收敛。我们进行了大量的数值实验来支持我们的理论发现。我们建议的方法的PYTHON实现可在https://github.com/leyuanheart/SEAL.上获得这篇文章的补充材料可以在网上找到。
Abstract We consider reinforcement learning (RL) methods in offline domains without additional online data collection, such as mobile health applications. Most of existing policy optimization algorithms in the computer science literature are developed in online settings where data are easy to collect or simulate. Their generalizations to mobile health applications with a pre-collected offline dataset remain are less explored. The aim of this article is to develop a novel advantage learning framework in order to efficiently use pre-collected data for policy optimization. The proposed method takes an optimal Q-estimator computed by any existing state-of-the-art RL algorithms as input, and outputs a new policy whose value is guaranteed to converge at a faster rate than the policy derived based on the initial Q-estimator. Extensive numerical experiments are conducted to back up our theoretical findings. A Python implementation of our proposed method is available at https://github.com/leyuanheart/SEAL. Supplementary materials for this article are available online.