An adaptive polyak heavy-ball method
An adaptive polyak heavy-ball method
复制标题
DOI：
10.1007/s10994-022-06215-7
复制
发表时间：
2022-07
期刊：
影响因子：
7.5
通讯作者：
S. Saab;S. Phoha;Minghui Zhu;A. Ray
中科院分区：
文献类型：
--
作者：
S. Saab;S. Phoha;Minghui Zhu;A. Ray
重球（HB）方法已成为解决大规模机器学习问题的著名实践，当目标函数是平滑且强凸的时，使用 Polyak 的最优超参数，它可以实现最快的局部收敛速度。然而，这种收敛速度是基于特定的不确定性和时不变的超参数，这限制了其潜力。在本文中，我们提出了一种自适应 HB，它可以在每次迭代时估计 Polyak 的最佳超参数。我们的自适应方法利用当前和先前模型参数及其梯度的绝对差异。这种表示形式允许计算高效的优化器。我们证明了我们的方法保证了平滑和强凸目标函数的全局线性收敛率。而在随机设置中，我们表明所提出的随机算法对于具有有界梯度的非凸平滑函数几乎肯定会收敛。我们验证了我们的方法在图像分类数据集上的有效性，无需经验调整，以及它在二次和非凸函数上的优越性，同时将其性能与最先进的优化器进行比较。
The heavy-ball (HB) method has become a well-known practice for large-scale machine learning problems, and it can achieve the fastest local convergence rate when objective functions are smooth and strongly convex using Polyak’s optimal hyper-parameters. However, such convergence rates are based on specific uncertain and time-invariant hyper-parameters that limit its potential. In this paper, we propose an adaptive HB that estimates the Polyak’s optimal hyper-parameters at each iteration. Our adaptive approach employs the absolute differences of current and previous model parameters and their gradients. Such representation allows for a computationally efficient optimizer. We show that our method guarantees a global linear convergence rate for smooth and strongly convex objective functions. Whereas in the stochastic setting, we show that proposed stochastic algorithm converges almost surely for non-convex smooth functions with bounded gradient. We validate the effectiveness of our method on image classification datasets with no empirical tuning, and its superiority on quadratic and non-convex functions while comparing its performance to the state-of-the-art optimizers.