Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundations
Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundations
复制标题
DOI：
--
复制
发表时间：
2018-11
期刊：
影响因子：
--
通讯作者：
Qianxiao Li;Cheng Tai;E. Weinan
中科院分区：
文献类型：
--
作者：
Qianxiao Li;Cheng Tai;E. Weinan
我们发展了用于分析随机梯度算法动力学的随机修正方程（SME）框架的数学基础，其中后者由一类具有小噪声参数的随机微分方程近似。我们证明了这种近似在数学上可以理解为一个弱近似，从而得到了随机梯度下降法（SGD）、动量SGD和随机Nesterov加速梯度法在一般随机目标设置下的近似的一些精确和有用的结果。我们还通过显式计算证明，这种连续时间方法可以揭示对所考虑的随机梯度算法的重要分析见解，这在纯粹的离散时间设置中可能不容易获得。
We develop the mathematical foundations of the stochastic modified equations (SME) framework for analyzing the dynamics of stochastic gradient algorithms, where the latter is approximated by a class of stochastic differential equations with small noise parameters. We prove that this approximation can be understood mathematically as an weak approximation, which leads to a number of precise and useful results on the approximations of stochastic gradient descent (SGD), momentum SGD and stochastic Nesterov's accelerated gradient method in the general setting of stochastic objectives. We also demonstrate through explicit calculations that this continuous-time approach can uncover important analytical insights into the stochastic gradient algorithms under consideration that may not be easy to obtain in a purely discrete-time setting.