Semigroups of stochastic gradient descent and online principal component analysis: properties and diffusion approximations
Semigroups of stochastic gradient descent and online principal component analysis: properties and diffusion approximations
复制标题
DOI：
10.4310/cms.2018.v16.n3.a8
复制
发表时间：
2017-12
影响因子：
1
通讯作者：
Yuanyuan Feng-;Lei Li;Jian‐Guo Liu
中科院分区：
文献类型：
--
作者：
Yuanyuan Feng-;Lei Li;Jian‐Guo Liu
我们研究了机器学习中两个重要算法的马尔可夫半群：随机梯度下降（SGD）和在线主成分分析（PCA）。我们研究了小跳跃对半群性质的影响。性质，包括正则性保持，$L^{\infty}$压缩进行了讨论。这些半群是概率发展半群的对偶，而后者是L^{1}$收缩的保正的。利用这些性质，我们证明了在$\mathbb{R}^d$（在球面$\mathbb{S}^{d-1}$上）中的随机微分方程（SDEs）可以弱近似SGD（在线PCA）。这些SDE可以用于提供这些算法的行为的一些见解。
We study the Markov semigroups for two important algorithms from machine learning: stochastic gradient descent (SGD) and online principal component analysis (PCA). We investigate the effects of small jumps on the properties of the semi-groups. Properties including regularity preserving, $L^{\infty}$ contraction are discussed. These semigroups are the dual of the semigroups for evolution of probability, while the latter are $L^{1}$ contracting and positivity preserving. Using these properties, we show that stochastic differential equations (SDEs) in $\mathbb{R}^d$ (on the sphere $\mathbb{S}^{d-1}$) can be used to approximate SGD (online PCA) weakly. These SDEs may be used to provide some insights of the behaviors of these algorithms.