Cooperative SGD: A Unified Framework for the Design and Analysis of Local-Update SGD Algorithms
Cooperative SGD: A Unified Framework for the Design and Analysis of Local-Update SGD Algorithms
复制标题
DOI：
--
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Jianyu Wang;Gauri Joshi
中科院分区：
文献类型：
--
作者：
Jianyu Wang;Gauri Joshi
当使用大量节点或大量边缘设备使用随机梯度下降（SGD）训练机器学习模型时，每次迭代同步梯度的通信成本是限制系统可扩展性并阻碍并行计算优势的关键瓶颈。本地更新SGD算法，即工作节点执行SGD的本地迭代并定期同步其本地模型，可以有效降低通信频率并节省通信延迟。在本文中，我们提出了一个名为 Cooperative SGD 的强大框架，它包含了各种局部更新 SGD 算法（例如局部 SGD、弹性平均 SGD 和分散并行 SGD），并提供统一的收敛分析。值得注意的是，协作 SGD 框架提供的统一收敛分析的特殊情况产生了 1）针对一般非凸目标的弹性平均 SGD 的首次收敛分析，以及 2）对先前局部 SGD 和分散并行 SGD 分析的改进。此外，我们设计了新的算法，例如具有重叠计算和通信的弹性平均 SGD，以及分散周期平均，这些算法在达到相同的训练损失时比基线快 4 倍或更多。
When training machine learning models using stochastic gradient descent (SGD) with a large number of nodes or massive edge devices, the communication cost of synchronizing gradients at every iteration is a key bottleneck that limits the scalability of the system and hinders the benefit of parallel computation. Local-update SGD algorithms, where worker nodes perform local iterations of SGD and periodically synchronize their local models, can effectively reduce the communication frequency and save the communication delay. In this paper, we propose a powerful framework, named Cooperative SGD, that subsumes a variety of local-update SGD algorithms (such as local SGD, elastic averaging SGD, and decentralized parallel SGD) and provides a unified convergence analysis. Notably, special cases of the unified convergence analysis provided by the cooperative SGD framework yield 1) the first convergence analysis of elastic averaging SGD for general non-convex objectives, and 2) improvements upon previous analyses of local SGD and decentralized parallel SGD. Moreover, we design new algorithms such as elastic averaging SGD with overlapped computation and communication, and decentralized periodic averaging which are shown to be 4x or more faster than the baseline in reaching the same training loss.