Adaptive Learning Rates with Maximum Variation Averaging
Adaptive Learning Rates with Maximum Variation Averaging
复制标题
DOI：
--
复制
发表时间：
2020-06
期刊：
影响因子：
--
通讯作者：
Chen Zhu;Yu Cheng;Zhe Gan;Furong Huang;Jingjing Liu;T. Goldstein
中科院分区：
文献类型：
--
作者：
Chen Zhu;Yu Cheng;Zhe Gan;Furong Huang;Jingjing Liu;T. Goldstein
自适应梯度方法如RMSProp和Adam使用平方梯度的指数移动估计来计算逐元素自适应步长并处理噪声梯度。然而，由于不稳定或极端的自适应学习率，Adam在某些问题中可能会有不期望的收敛行为。人们已经提出了AMSGrad和AdaBound等方法来稳定Adam在训练后期的自适应学习率，但在训练Transformers等一些实际任务中，它们并没有超过Adam。在本文中，我们提出了一种自适应学习率规则，其中运行均方梯度被替换为加权平均值，选择权重以最大化每个坐标的估计方差。这给出了局部梯度方差的最坏情况估计，当存在大曲率或噪声梯度时采取较小的步骤，从而产生比Adam更理想的收敛行为。我们分析并展示了我们的自适应平均方法在图像分类，神经机器翻译和自然语言理解任务上的改进功效。
Adaptive gradient methods such as RMSProp and Adam use exponential moving estimate of the squared gradient to compute element-wise adaptive step sizes and handle noisy gradients. However, Adam can have undesirable convergence behavior in some problems due to unstable or extreme adaptive learning rates. Methods such as AMSGrad and AdaBound have been proposed to stabilize the adaptive learning rates of Adam in the later stage of training, but they do not outperform Adam in some practical tasks such as training Transformers. In this paper, we propose an adaptive learning rate rule in which the running mean squared gradient is replaced by a weighted mean, with weights chosen to maximize the estimated variance of each coordinate. This gives a worst-case estimate for the local gradient variance, taking smaller steps when large curvatures or noisy gradients are present, resulting in more desirable convergence behavior than Adam. We analyze and demonstrate the improved efficacy of our adaptive averaging approach on image classification, neural machine translation and natural language understanding tasks.