Analysis of Q-learning with Adaptation and Momentum Restart for Gradient Descent
Analysis of Q-learning with Adaptation and Momentum Restart for Gradient Descent
复制标题
DOI：
10.24963/ijcai.2020/422
复制
发表时间：
2020-07
期刊：
影响因子：
--
通讯作者：
Chuhan Wu;Fangzhao Wu;Tao Qi;Yongfeng Huang
中科院分区：
文献类型：
--
作者：
Chuhan Wu;Fangzhao Wu;Tao Qi;Yongfeng Huang
现有的Q-learning收敛性分析主要集中在随机梯度下降（SGD）类型的更新上。尽管自适应矩估计（Adam）已被广泛用于实际的Q-learning算法中，但对于这种类型的更新，Q-learning并没有任何收敛保证。本文首先表征了Q-AMSGrad的收敛速度，这是一种带有AMSGrad更新的q -学习算法（理论分析中常用的Adam替代算法）。为了进一步提高性能，我们提出在Q-AMSGrad中加入动量重启方案，从而产生所谓的Q-AMSGradR算法。并给出了Q-AMSGradR的收敛速度。我们在线性二次型调节器问题上的实验表明，这两种提出的q -学习算法优于具有SGD更新的普通q -学习算法。在一批Atari 2600游戏中，这两种算法也比DQN学习方法表现出明显更好的性能。
Existing convergence analyses of Q-learning mostly focus on the vanilla stochastic gradient descent (SGD) type of updates. Despite the Adaptive Moment Estimation (Adam) has been commonly used for practical Q-learning algorithms, there has not been any convergence guarantee provided for Q-learning with such type of updates. In this paper, we first characterize the convergence rate for Q-AMSGrad, which is the Q-learning algorithm with AMSGrad update (a commonly adopted alternative of Adam for theoretical analysis). To further improve the performance, we propose to incorporate the momentum restart scheme to Q-AMSGrad, resulting in the so-called Q-AMSGradR algorithm. The convergence rate of Q-AMSGradR is also established. Our experiments on a linear quadratic regulator problem demonstrate that the two proposed Q-learning algorithms outperform the vanilla Q-learning with SGD updates. The two algorithms also exhibit significantly better performance than the DQN learning method over a batch of Atari 2600 games.