Multiscale Analysis of Accelerated Gradient Methods
Multiscale Analysis of Accelerated Gradient Methods
复制标题
加速梯度法的多尺度分析
DOI：
--
复制
发表时间：
2018
影响因子：
3.1
通讯作者：
M. Farazmand
中科院分区：
文献类型：
--
作者：
M. Farazmand
加速梯度下降迭代在优化中有着广泛的应用。众所周知，在连续时间的限制，这些迭代收敛到一个二阶微分方程，我们称之为加速梯度流。利用几何奇异摄动理论，证明了在一定条件下，加速梯度流具有一个吸引不变的慢流形，流的轨迹渐近收敛于该慢流形.我们得到一个一般的明确表达的形式的功能系列的扩展，近似的慢流形的任何任意顺序的准确性。对于首阶，退化到这个慢流形的加速梯度流与通常的梯度下降相一致。我们说明了我们的研究结果的影响三个例子。
Accelerated gradient descent iterations are widely used in optimization. It is known that, in the continuous-time limit, these iterations converge to a second-order differential equation which we refer to as the accelerated gradient flow. Using geometric singular perturbation theory, we show that, under certain conditions, the accelerated gradient flow possesses an attracting invariant slow manifold to which the trajectories of the flow converge asymptotically. We obtain a general explicit expression in the form of functional series expansions that approximates the slow manifold to any arbitrary order of accuracy. To the leading order, the accelerated gradient flow reduced to this slow manifold coincides with the usual gradient descent. We illustrate the implications of our results on three examples.