Generalized Momentum-Based Methods: A Hamiltonian Perspective
Generalized Momentum-Based Methods: A Hamiltonian Perspective
复制标题
DOI：
10.1137/20m1322716
复制
发表时间：
2019-06
期刊：
影响因子：
--
通讯作者：
Jelena Diakonikolas;Michael I. Jordan
中科院分区：
文献类型：
--
作者：
Jelena Diakonikolas;Michael I. Jordan
我们采用基于汉密尔顿的观点来概括Nesterov的加速梯度下降，而Polyak的重球方法则可以在欧几里得和非欧几里德规范矢量空间（可能）最小化的（可能）最小化的情况下进行广泛的动量方法。我们的观点导致了这些方法在函数值（在凸优化的设置中）和梯度（在不受限制的，可能是非凸的，优化的设置）中，对这些方法的收敛性进行了通用和统一的非肌电分析。我们的方法依赖于随着时变的哈密顿量，该哈密顿量产生广义动量方法作为运动方程。这些方法的收敛分析是直观的，并且基于时间依赖性的哈密顿量的保守量。
We take a Hamiltonian-based perspective to generalize Nesterov's accelerated gradient descent and Polyak's heavy ball method to a broad class of momentum methods in the setting of (possibly) constrained minimization in Euclidean and non-Euclidean normed vector spaces. Our perspective leads to a generic and unifying nonasymptotic analysis of convergence of these methods in both the function value (in the setting of convex optimization) and in norm of the gradient (in the setting of unconstrained, possibly nonconvex, optimization). Our approach relies upon a time-varying Hamiltonian that produces generalized momentum methods as its equations of motion. The convergence analysis for these methods is intuitive and is based on the conserved quantities of the time-dependent Hamiltonian.