On The Convergence Of Policy Iteration-Based Reinforcement Learning With Monte Carlo Policy Evaluation-猫眼课题宝

DOI：

10.48550/arxiv.2301.09709

发表时间：

2023-01

期刊：

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Anna Winnicki;R. Srikant

强化学习中的常用技术是评估给定策略的蒙特卡罗模拟的价值函数，并使用估计价值函数来获得相对于估计价值函数贪婪的新策略。在这种情况下，一个众所周知的长期悬而未决的问题是，当根据从实施政策获得的单个样本路径收集的数据来估计政策的价值函数时，证明这种方案的收敛性（参见[Sutton and Barto，2018]第99页，[Tsitsiklis，2002]第8页）。我们通过证明这种策略迭代方案的首次访问版本确实收敛到最优策略来提出开放问题的解决方案，前提是策略改进步骤使用前瞻[Silver et al., 2016, Mnih et al., 2016, Silver et al., 2017b]而不是简单的贪婪策略改进。我们在表格设置中提供了原始开放问题的结果，并且还提供了函数逼近设置的扩展，其中我们表明算法产生的策略在函数逼近误差内执行接近最优策略。

A common technique in reinforcement learning is to evaluate the value function from Monte Carlo simulations of a given policy, and use the estimated value function to obtain a new policy which is greedy with respect to the estimated value function. A well-known longstanding open problem in this context is to prove the convergence of such a scheme when the value function of a policy is estimated from data collected from a single sample path obtained from implementing the policy (see page 99 of [Sutton and Barto, 2018], page 8 of [Tsitsiklis, 2002]). We present a solution to the open problem by showing that a first-visit version of such a policy iteration scheme indeed converges to the optimal policy provided that the policy improvement step uses lookahead [Silver et al., 2016, Mnih et al., 2016, Silver et al., 2017b] rather than a simple greedy policy improvement. We provide results both for the original open problem in the tabular setting and also present extensions to the function approximation setting, where we show that the policy resulting from the algorithm performs close to the optimal policy within a function approximation error.

On The Convergence Of Policy Iteration-Based Reinforcement Learning With Monte Carlo Policy Evaluation

文献摘要

相似文献