On the Asymptotic Convergence and Acceleration of Gradient Methods
On the Asymptotic Convergence and Acceleration of Gradient Methods
复制标题
DOI：
10.1007/s10915-021-01685-8
复制
发表时间：
2019-08
影响因子：
2.5
通讯作者：
Yakui Huang;Yuhong Dai;Xinwei Liu;Hongchao Zhang
中科院分区：
文献类型：
--
作者：
Yakui Huang;Yuhong Dai;Xinwei Liu;Hongchao Zhang
我们考虑一系列梯度方法的渐近行为，其中包括最速下降和最小梯度方法作为特殊实例。事实证明，该族中的每种方法都会在两个方向之间渐近之字形。还给出了目标值、梯度范数和步长的渐近收敛结果。为了加速梯度方法系列，我们进一步利用步长的光谱特性来打破之字形模式。特别是，通过对最小化二维严格凸二次函数施加有限终止来导出新的步长。结果表明，对于一般二次函数，所提出的步长渐近收敛于 Hessian 矩阵最大特征值的倒数。此外，基于这种光谱特性，我们提出了一种结合 Barzilai-Borwein 方法的周期性梯度方法。与最近一些成功的梯度方法的数值比较表明我们的新方法非常有前途。
We consider the asymptotic behavior of a family of gradient methods, which include the steepest descent and minimal gradient methods as special instances. It is proved that each method in the family will asymptotically zigzag between two directions. Asymptotic convergence results of the objective value, gradient norm, and stepsize are presented as well. To accelerate the family of gradient methods, we further exploit spectral properties of stepsizes to break the zigzagging pattern. In particular, a new stepsize is derived by imposing finite termination on minimizing two-dimensional strictly convex quadratic function. It is shown that, for the general quadratic function, the proposed stepsize asymptotically converges to the reciprocal of the largest eigenvalue of the Hessian. Furthermore, based on this spectral property, we propose a periodic gradient method by incorporating the Barzilai-Borwein method. Numerical comparisons with some recent successful gradient methods show that our new method is very promising.