Conformal symplectic and relativistic optimization
Conformal symplectic and relativistic optimization
复制标题
DOI：
10.1088/1742-5468/abcaee
复制
发表时间：
2019-03
期刊：
影响因子：
--
通讯作者：
G. Francca;Jeremias Sulam;Daniel P. Robinson;R. Vidal
中科院分区：
文献类型：
--
作者：
G. Francca;Jeremias Sulam;Daniel P. Robinson;R. Vidal
可以说，机器学习中最流行的两种加速或基于动量的优化方法是Nesterov的加速梯度和Polyaks的重球，两者都对应于特定二阶摩擦微分方程的不同离散化。这种与连续时间动力系统的联系有助于揭开优化中加速现象的神秘面纱。在这里，我们研究了一类耗散（共形）哈密顿系统的保结构离散化，使我们能够分析Nesterov和重球的辛结构，除了这些方法提供了一些新的见解。此外，我们提出了一个新的算法的基础上的耗散相对论系统，归一化的动量，并可能导致更稳定/更快的优化。重要的是，这种方法推广了涅斯捷罗夫和重球，每一个都被恢复为不同的限制情况，并且在没有额外成本的情况下具有潜在的优势。
Arguably, the two most popular accelerated or momentum-based optimization methods in machine learning are Nesterov’s accelerated gradient and Polyaks’s heavy ball, both corresponding to different discretizations of a particular second order differential equation with friction. Such connections with continuous-time dynamical systems have been instrumental in demystifying acceleration phenomena in optimization. Here we study structure-preserving discretizations for a certain class of dissipative (conformal) Hamiltonian systems, allowing us to analyse the symplectic structure of both Nesterov and heavy ball, besides providing several new insights into these methods. Moreover, we propose a new algorithm based on a dissipative relativistic system that normalizes the momentum and may result in more stable/faster optimization. Importantly, such a method generalizes both Nesterov and heavy ball, each being recovered as distinct limiting cases, and has potential advantages at no additional cost.