Overlap Local-SGD: An Algorithmic Approach to Hide Communication Delays in Distributed SGD
Overlap Local-SGD: An Algorithmic Approach to Hide Communication Delays in Distributed SGD
复制标题
DOI：
10.1109/icassp40776.2020.9053834
复制
发表时间：
2020-02
期刊：
影响因子：
--
通讯作者：
Jianyu Wang;Hao Liang;Gauri Joshi
中科院分区：
文献类型：
--
作者：
Jianyu Wang;Hao Liang;Gauri Joshi
分布式随机梯度下降（SGD）对于将机器学习算法扩展到大量计算节点至关重要。然而，基础设施的变化，如高通信延迟或随机节点减速极大地阻碍了分布式SGD算法的性能，特别是在无线系统或传感器网络。在本文中，我们提出了一种算法的方法命名为重叠本地SGD（及其动量变体）重叠通信和计算，以加快分布式训练过程。这种方法也有助于减轻离散效应。我们通过在每个节点上添加一个锚模型来实现这一点。在多次本地更新之后，本地训练的模型将被拉回到同步的锚模型，而不是与其他模型通信。在CIFAR-10数据集上训练深度神经网络的实验结果证明了重叠局部SGD的有效性。我们还提供了一个收敛性保证所提出的算法下的非凸目标函数。本文的完整版本与额外的例子和证明可访问：http://andrew.cmu.edu/user/gaurij/overlap_local_SGD.pdf。
Distributed stochastic gradient descent (SGD) is essential for scaling the machine learning algorithms to a large number of computing nodes. However, the infrastructures variability such as high communication delay or random node slowdown greatly impedes the performance of distributed SGD algorithm, especially in a wireless system or sensor networks. In this paper, we propose an algorithmic approach named Overlap Local-SGD (and its momentum variant) to overlap communication and computation so as to speedup the distributed training procedure. The approach can help to mitigate the straggler effects as well. We achieve this by adding an anchor model on each node. After multiple local updates, locally trained models will be pulled back towards the synchronized anchor model rather than communicating with others. Experimental results of training a deep neural network on CIFAR-10 dataset demonstrate the effectiveness of Overlap Local-SGD. We also provide a convergence guarantee for the proposed algorithm under non-convex objective functions.A full version of this paper with additional examples and proofs is accessible at: http://andrew.cmu.edu/user/gaurij/overlap_local_SGD.pdf.