Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity
Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity
复制标题
DOI：
--
复制
发表时间：
2022-02
期刊：
影响因子：
--
通讯作者：
Laixi Shi;Gen Li;Yuting Wei;Yuxin Chen;Yuejie Chi
中科院分区：
文献类型：
--
作者：
Laixi Shi;Gen Li;Yuting Wei;Yuxin Chen;Yuejie Chi
离线或批量强化学习旨在使用历史数据来学习接近最优的策略，而无需主动探索环境。为了解决许多离线数据集覆盖范围不足和样本稀缺的问题，最近引入了悲观主义原则来减轻估计值的高偏差。虽然基于模型的算法的悲观变体（例如，具有较低置信界限的值迭代）已经在理论上进行了研究，但它们的无模型算法（不需要显式模型估计）尚未得到充分研究，特别是在样本效率方面。为了解决这个不足，我们在有限水平马尔可夫决策过程的背景下研究了 Q 学习的悲观变体，并在不需要完全覆盖状态动作空间的单策略集中性假设下表征其样本复杂性。此外，还提出了一种方差减少的悲观 Qlearning 算法来实现接近最优的样本复杂度。总而言之，这项工作凸显了离线强化学习中无模型算法与悲观主义和方差减少结合使用时的效率。
Offline or batch reinforcement learning seeks to learn a near-optimal policy using history data without active exploration of the environment. To counter the insufficient coverage and sample scarcity of many offline datasets, the principle of pessimism has been recently introduced to mitigate high bias of the estimated values. While pessimistic variants of model-based algorithms (e.g., value iteration with lower confidence bounds) have been theoretically investigated, their modelfree counterparts — which do not require explicit model estimation — have not been adequately studied, especially in terms of sample efficiency. To address this inadequacy, we study a pessimistic variant of Q-learning in the context of finitehorizon Markov decision processes, and characterize its sample complexity under the single-policy concentrability assumption which does not require the full coverage of the state-action space. In addition, a variance-reduced pessimistic Qlearning algorithm is proposed to achieve nearoptimal sample complexity. Altogether, this work highlights the efficiency of model-free algorithms in offline RL when used in conjunction with pessimism and variance reduction.