ADASECANT: Robust Adaptive Secant Method for Stochastic Gradient
ADASECANT: Robust Adaptive Secant Method for Stochastic Gradient
复制标题
ADASECANT：随机梯度的鲁棒自适应割线法
DOI：
--
复制
发表时间：
2014
期刊：
影响因子：
--
通讯作者：
Yoshua Bengio
中科院分区：
文献类型：
--
作者：
Çaglar Gülçehre;Yoshua Bengio
随机梯度算法一直是大规模学习问题的主要焦点，并在机器学习方面取得了重要的成功。SGD的收敛性取决于仔细选择学习率和随机估计梯度中的噪声量。在本文中，我们提出了一种新的自适应学习率算法，该算法利用曲率信息自动调整学习率。损失函数的元曲率信息由随机一阶梯度的局部统计量估计。我们进一步提出了一种新的方差缩减技术来加快收敛速度。在深度神经网络的初步实验中，与流行的随机梯度算法相比，我们获得了更好的性能。
Stochastic gradient algorithms have been the main focus of large-scale learning problems and they led to important successes in machine learning. The convergence of SGD depends on the careful choice of learning rate and the amount of the noise in stochastic estimates of the gradients. In this paper, we propose a new adaptive learning rate algorithm, which utilizes curvature information for automatically tuning the learning rates. The information about the element-wise curvature of the loss function is estimated from the local statistics of the stochastic first order gradients. We further propose a new variance reduction technique to speed up the convergence. In our preliminary experiments with deep neural networks, we obtained better performance compared to the popular stochastic gradient algorithms.