PAGE-PG: A Simple and Loopless Variance-Reduced Policy Gradient Method with Probabilistic Gradient Estimation
PAGE-PG: A Simple and Loopless Variance-Reduced Policy Gradient Method with Probabilistic Gradient Estimation
复制标题
DOI：
--
复制
发表时间：
2022-02
期刊：
影响因子：
--
通讯作者：
Matilde Gargiani;Andrea Zanelli;Andrea Martinelli;T. Summers;J. Lygeros
中科院分区：
文献类型：
--
作者：
Matilde Gargiani;Andrea Zanelli;Andrea Martinelli;T. Summers;J. Lygeros
尽管策略梯度方法取得了成功，但它们的梯度估计方差很高，这可能导致样本复杂度不令人满意。最近，许多方差减少扩展的政策梯度方法，证明更好的样本复杂性和竞争力的数值性能已被提出。在对一些主要的方差降低的REINFORCE型方法进行了紧凑的调查之后，我们提出了用于策略梯度的概率梯度估计（PAGE-PG），这是一种基于两种更新类型之间的概率切换的新型无环方差降低策略梯度方法。我们的方法受到监督学习的PAGE估计器的启发，并利用重要性采样来获得无偏梯度估计器。我们发现，PAGE-PG享有$\mathcal{O}\left（\mathcal ^{-3} \right）$平均样本复杂性，以达到$\mathcal $-平稳的解决方案，这匹配的样本复杂性，其最具竞争力的同行在相同的设置。数值评估证实了我们的方法对经典控制任务的竞争力。
Despite their success, policy gradient methods suffer from high variance of the gradient estimate, which can result in unsatisfactory sample complexity. Recently, numerous variance-reduced extensions of policy gradient methods with provably better sample complexity and competitive numerical performance have been proposed. After a compact survey on some of the main variance-reduced REINFORCE-type methods, we propose ProbAbilistic Gradient Estimation for Policy Gradient (PAGE-PG), a novel loopless variance-reduced policy gradient method based on a probabilistic switch between two types of updates. Our method is inspired by the PAGE estimator for supervised learning and leverages importance sampling to obtain an unbiased gradient estimator. We show that PAGE-PG enjoys a $\mathcal{O}\left( \epsilon^{-3} \right)$ average sample complexity to reach an $\epsilon$-stationary solution, which matches the sample complexity of its most competitive counterparts under the same setting. A numerical evaluation confirms the competitive performance of our method on classical control tasks.