CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guarantee
CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guarantee
复制标题
DOI：
--
复制
发表时间：
2020-11
期刊：
影响因子：
--
通讯作者：
Tengyu Xu;Yingbin Liang;Guanghui Lan
中科院分区：
文献类型：
--
作者：
Tengyu Xu;Yingbin Liang;Guanghui Lan
在安全强化学习（SRL）问题中，智能体探索环境以最大化预期总回报，同时避免违反对一些预期总成本的某些约束。一般来说，这样的SRL问题有非凸目标函数受到多个非凸约束，因此是非常具有挑战性的解决，特别是提供一个全局最优的政策。许多流行的SRL算法采用原始-对偶结构，利用对偶变量的更新来满足约束。相比之下，我们提出了一个原始的方法，称为约束纠正策略优化（CRPO），更新的政策之间的目标改进和约束满足交替。CRPO提供了一个原始类型的算法框架来解决SRL问题，其中每个策略更新可以采取任何策略优化步骤的变体。为了证明CRPO的理论性能，我们采用自然的政策梯度（NPG）的每个政策更新步骤，并表明CRPO实现了$\mathcal{O}（1/\sqrt{T}）$收敛速度的全局最优政策的约束政策集和$\mathcal{O}（1/\sqrt{T}）$约束满足的错误界。这是第一次有限时间分析的原始SRL算法的全局最优性保证。我们的实证结果表明，CRPO可以优于现有的原始-对偶基线算法显着。
In safe reinforcement learning (SRL) problems, an agent explores the environment to maximize an expected total reward and meanwhile avoids violation of certain constraints on a number of expected total costs. In general, such SRL problems have nonconvex objective functions subject to multiple nonconvex constraints, and hence are very challenging to solve, particularly to provide a globally optimal policy. Many popular SRL algorithms adopt a primal-dual structure which utilizes the updating of dual variables for satisfying the constraints. In contrast, we propose a primal approach, called constraint-rectified policy optimization (CRPO), which updates the policy alternatingly between objective improvement and constraint satisfaction. CRPO provides a primal-type algorithmic framework to solve SRL problems, where each policy update can take any variant of policy optimization step. To demonstrate the theoretical performance of CRPO, we adopt natural policy gradient (NPG) for each policy update step and show that CRPO achieves an $\mathcal{O}(1/\sqrt{T})$ convergence rate to the global optimal policy in the constrained policy set and an $\mathcal{O}(1/\sqrt{T})$ error bound on constraint satisfaction. This is the first finite-time analysis of primal SRL algorithms with global optimality guarantee. Our empirical results demonstrate that CRPO can outperform the existing primal-dual baseline algorithms significantly.