Warm-Start Actor-Critic: From Approximation Error to Sub-optimality Gap
Warm-Start Actor-Critic: From Approximation Error to Sub-optimality Gap
复制标题
DOI：
10.48550/arxiv.2306.11271
复制
发表时间：
2023-06
期刊：
影响因子：
--
通讯作者：
Hang Wang;Sen Lin;Junshan Zhang
中科院分区：
文献类型：
--
作者：
Hang Wang;Sen Lin;Junshan Zhang
热启动强化学习（RL），通过离线训练获得的先验策略的帮助下，正在成为一个有前途的RL方法的实际应用。最近的实证研究表明，热启动RL的性能可以提高\textit{快速}在某些情况下，但在其他情况下，变得\textit{停滞}，特别是当使用的函数近似。为此，这项工作的主要目标是建立对“\textit{是否以及何时可以通过离线RL的热启动政策显着加速在线学习？}的基本理解''.具体来说，我们考虑了广泛使用的演员-评论家（A-C）的方法与事先的政策。我们首先分别量化Actor更新和Critic更新中的近似误差。接下来，我们将Warm-Start A-C算法转换为具有扰动的牛顿方法，并研究近似误差对不准确Actor/Critic更新的有限时间学习性能的影响。在一般的技术条件下，我们得到的上界，这揭示了实现所需的有限学习性能的温启动A-C算法。特别是，我们的研究结果表明，减少在线学习中的算法偏差至关重要。我们还获得了Warm-Start A-C算法的次最优性差距的下界，以量化偏差和误差传播的影响。
Warm-Start reinforcement learning (RL), aided by a prior policy obtained from offline training, is emerging as a promising RL approach for practical applications. Recent empirical studies have demonstrated that the performance of Warm-Start RL can be improved \textit{quickly} in some cases but become \textit{stagnant} in other cases, especially when the function approximation is used. To this end, the primary objective of this work is to build a fundamental understanding on ``\textit{whether and when online learning can be significantly accelerated by a warm-start policy from offline RL?}''. Specifically, we consider the widely used Actor-Critic (A-C) method with a prior policy. We first quantify the approximation errors in the Actor update and the Critic update, respectively. Next, we cast the Warm-Start A-C algorithm as Newton's method with perturbation, and study the impact of the approximation errors on the finite-time learning performance with inaccurate Actor/Critic updates. Under some general technical conditions, we derive the upper bounds, which shed light on achieving the desired finite-learning performance in the Warm-Start A-C algorithm. In particular, our findings reveal that it is essential to reduce the algorithm bias in online learning. We also obtain lower bounds on the sub-optimality gap of the Warm-Start A-C algorithm to quantify the impact of the bias and error propagation.