Parallel and distributed asynchronous adaptive stochastic gradient methods
Parallel and distributed asynchronous adaptive stochastic gradient methods
复制标题
DOI：
10.1007/s12532-023-00237-5
复制
发表时间：
2020-02
影响因子：
6.3
通讯作者：
Yangyang Xu;Yibo Xu;Yonggui Yan;Colin Sutcher-Shepard;Leopold Grinberg;Jiewei Chen
中科院分区：
文献类型：
--
作者：
Yangyang Xu;Yibo Xu;Yonggui Yan;Colin Sutcher-Shepard;Leopold Grinberg;Jiewei Chen
随机梯度方法（SGM）是训练深度学习模型的主要方法。自适应版本（例如，Adam和AMSGrad）在实践中得到了广泛的使用，部分原因是它们比非自适应版本实现了更快的收敛，同时产生的开销很少。另一方面，异步（异步）并行计算表现出显着更高的速度比同步（同步）对应。异步并行非自适应SGM已经在文献中从理论和实际性能的角度进行了很好的研究。自适应SGM也可以以并行方式实现，没有太大困难。然而，据我们所知，没有理论结果的并行自适应SGM已建立。自适应SGM的二阶矩项是自适应SGM分析的难点。在本文中，我们提出了一种基于AMSGrad的并行自适应SGM。我们表明，所提出的方法继承了收敛保证AMSGrad的凸和非凸问题，如果老化（也称为延迟）所造成的dupy是有界的。我们的收敛速度结果表明，如果，其中是陈旧性，K是迭代次数，则并行化速度几乎是线性的。该方法在凸和非凸机器学习问题上进行了测试，数值结果表明其明显优于同步对应和并行非自适应SGM。我们的代码已在https://github.com/RPI-OPT/APAM上发布。
Stochastic gradient methods (SGMs) are the predominant approaches to train deep learning models. The adaptive versions (e.g., Adam and AMSGrad) have been extensively used in practice, partly because they achieve faster convergence than the non-adaptive versions while incurring little overhead. On the other hand, asynchronous (async) parallel computing has exhibited significantly higher speed-up over its synchronous (sync) counterpart. Async-parallel non-adaptive SGMs have been well studied in the literature from the perspectives of both theory and practical performance. Adaptive SGMs can also be implemented without much difficulty in an async-parallel way. However, to the best of our knowledge, no theoretical result of async-parallel adaptive SGMs has been established. The difficulty for analyzing adaptive SGMs with async updates originates from the second moment term. In this paper, we propose an async-parallel adaptive SGM based on AMSGrad. We show that the proposed method inherits the convergence guarantee of AMSGrad for both convex and non-convex problems, if the staleness (also called delay) caused by asynchrony is bounded. Our convergence rate results indicate a nearly linear parallelization speed-up if, whereis the staleness andKis the number of iterations. The proposed method is tested on both convex and non-convex machine learning problems, and the numerical results demonstrate its clear advantages over the sync counterpart and the async-parallel nonadaptive SGM. Our code has been released at https://github.com/RPI-OPT/APAM.