Policy Gradient in Robust MDPs with Global Convergence Guarantee
Policy Gradient in Robust MDPs with Global Convergence Guarantee
复制标题
DOI：
--
复制
发表时间：
2022-12
期刊：
影响因子：
--
通讯作者：
Qiuhao Wang;C. Ho;Marek Petrik
中科院分区：
文献类型：
--
作者：
Qiuhao Wang;C. Ho;Marek Petrik
鲁棒马尔可夫决策过程（RMDPs）提供了一个很有前途的框架，计算可靠的政策，面对模型错误。许多成功的强化学习算法都建立在策略梯度方法的基础上，但使这些方法适应RMDPs一直是一个挑战。因此，RMDPs对大型实用领域的适用性仍然有限。本文提出了一种新的双环鲁棒策略梯度（DRPG），第一个通用的策略梯度方法RMDPs。与现有的鲁棒策略梯度算法相比，DRPG单调地减少近似误差，以保证收敛到全局最优策略表RMDPs。我们引入了一种新的参数化转移核，并通过基于梯度的方法求解内环鲁棒策略。最后，我们的数值结果证明了我们的新算法的实用性，并确认其全局收敛性。
Robust Markov decision processes (RMDPs) provide a promising framework for computing reliable policies in the face of model errors. Many successful reinforcement learning algorithms build on variations of policy-gradient methods, but adapting these methods to RMDPs has been challenging. As a result, the applicability of RMDPs to large, practical domains remains limited. This paper proposes a new Double-Loop Robust Policy Gradient (DRPG), the first generic policy gradient method for RMDPs. In contrast with prior robust policy gradient algorithms, DRPG monotonically reduces approximation errors to guarantee convergence to a globally optimal policy in tabular RMDPs. We introduce a novel parametric transition kernel and solve the inner loop robust policy via a gradient-based method. Finally, our numerical results demonstrate the utility of our new algorithm and confirm its global convergence properties.