Policy mirror descent for reinforcement learning: linear convergence, new sampling complexity, and generalized problem classes
Policy mirror descent for reinforcement learning: linear convergence, new sampling complexity, and generalized problem classes
复制标题
DOI：
10.1007/s10107-022-01816-5
复制
发表时间：
2021-01
影响因子：
2.7
通讯作者：
Guanghui Lan
中科院分区：
文献类型：
--
作者：
Guanghui Lan
我们提出了新的策略镜像下降（PMD）方法来解决具有强凸或一般凸正则器的强化学习（RL）问题。通过研究这些整体高度非凸问题的结构性质，我们表明PMD方法具有快速的线性收敛到全局最优性。我们发展了这些方法的随机对应物，并建立了一种新的方法。，)的采样复杂度来解决这些RL问题。（一般）使用不同采样方案的凸正则化器，其中注意目标精度。我们进一步证明了计算这些正则化器梯度的复杂度，如果必要的话，可以用（resp）来限定。，)的问题与强烈(尊重。（一般）凸正则化器。遗传记录了折现因子。据我们所知，这些复杂性界限，以及我们的算法发展，在优化和强化学习文献中似乎都是新的。这些凸正则化器的引入也极大地增强了灵活性，从而扩展了RL模型的适用性。
We present new policy mirror descent (PMD) methods for solving reinforcement learning (RL) problems with either strongly convex or general convex regularizers. By exploring the structural properties of these overall highly nonconvex problems we show that the PMD methods exhibit fast linear rate of convergence to the global optimality. We develop stochastic counterparts of these methods, and establish an(resp.,) sampling complexity for solving these RL problems with strongly (resp., general) convex regularizers using different sampling schemes, wheredenote the target accuracy. We further show that the complexity for computing the gradients of these regularizers, if necessary, can be bounded by(resp.,) for problems with strongly (resp., general) convex regularizers. Heredenotes the discounting factor. To the best of our knowledge, these complexity bounds, along with our algorithmic developments, appear to be new in both optimization and RL literature. The introduction of these convex regularizers also greatly enhances the flexibility and thus expands the applicability of RL models.