Momentum-Based Variance-Reduced Proximal Stochastic Gradient Method for Composite Nonconvex Stochastic Optimization
Momentum-Based Variance-Reduced Proximal Stochastic Gradient Method for Composite Nonconvex Stochastic Optimization
复制标题
DOI：
10.1007/s10957-022-02132-w
复制
发表时间：
2020-05
影响因子：
1.9
通讯作者：
Yangyang Xu;Yibo Xu
中科院分区：
文献类型：
--
作者：
Yangyang Xu;Yibo Xu
随机梯度方法（SGM）已被广泛用于解决随机问题或大规模机器学习问题。最近的作品采用各种技术来提高SGMs的收敛速度为凸和非凸的情况下。它们中的大多数在改进的SGM的部分或全部迭代中需要大量的样本。在本文中，我们提出了一个新的SGM，命名为PStorm，解决非凸非光滑随机问题。通过基于动量的方差缩减技术，如果均方平滑条件成立，PStorm可以实现最佳复杂度结果以产生随机平稳解。与现有的优化方法不同，PStorm算法在每次更新时只需使用一个或O（1）个样本就能达到优化结果。凭借此属性，PStorm可应用于在线学习问题，这些问题有利于基于一个或O（1）个新观察结果做出实时决策。此外，对于大规模的机器学习问题，PStorm可以通过小批量训练比其他需要大批量训练的优化方法和普通SGM更好地泛化，正如我们在训练稀疏全连接神经网络和稀疏卷积神经网络时所展示的那样。
Stochastic gradient methods (SGMs) have been extensively used for solving stochastic problems or large-scale machine learning problems. Recent works employ various techniques to improve the convergence rate of SGMs for both convex and nonconvex cases. Most of them require a large number of samples in some or all iterations of the improved SGMs. In this paper, we propose a new SGM, named PStorm, for solving nonconvex nonsmooth stochastic problems. With a momentum-based variance reduction technique, PStorm can achieve the optimal complexity resultto produce a stochastic-stationary solution, if a mean-squared smoothness condition holds. Different from existing optimal methods, PStorm can achieve theresult by using only one orO(1) samples in every update. With this property, PStorm can be applied to online learning problems that favor real-time decisions based on one orO(1) new observations. In addition, for large-scale machine learning problems, PStorm can generalize better by small-batch training than other optimal methods that require large-batch training and the vanilla SGM, as we demonstrate on training a sparse fully-connected neural network and a sparse convolutional neural network.