O(1) Communication for Distributed SGD through Two-Level Gradient Averaging
O(1) Communication for Distributed SGD through Two-Level Gradient Averaging
复制标题
DOI：
10.1109/cluster48925.2021.00054
复制
发表时间：
2020-06
期刊：
影响因子：
--
通讯作者：
Subhadeep Bhattacharya;Weikuan Yu;Fahim Chowdhury
中科院分区：
文献类型：
--
作者：
Subhadeep Bhattacharya;Weikuan Yu;Fahim Chowdhury
大型神经网络模型对分布式随机梯度下降（SGD）提出了巨大的通信挑战，对于n个参数的模型，每个工作者的每次迭代通信复杂度为$\mathcal{O}（n）$。已经提出了许多稀疏化和量化技术来压缩梯度，其中一些将每次迭代的通信复杂度降低到$\mathcal{O}（k）$，其中$k\ll n$。在本文中，我们引入了一种称为两级梯度平均（A2 SGD）的策略，在计算更新模型的两个全局平均值之前，将所有梯度合并为每个工人的两个局部平均值。A2 SGD还保留了局部误差，以保持快速收敛的方差。我们的分析表明，A2 SGD收敛类似于默认的分布式SGD算法。我们的评估验证了结论，并表明，A2 SGD显着降低了每个工人的通信流量，并提高了LSTM-PTB的整体培训时间由$3.2\times$和$23.2\times$，相比Top-K和QSGD，分别。我们使用两种优化器，SGD和亚当评估我们的方法的有效性。此外，我们对各种通信选项的评估表明，我们的方法在减少通信和融合方面都具有优势。据我们所知，A2 SGD是第一个实现每个工作者$\mathcal{O}$（1）通信复杂度而不会导致DNN模型的准确性显着下降，同时仅为分布式SGD通信每个工作者表示梯度的两个标量。
Large neural network models present a hefty communication challenge to distributed Stochastic Gradient Descent (SGD), with a per-iteration communication complexity of $\mathcal{O}(n)$ per worker for a model of n parameters. Many sparsification and quantization techniques have been proposed to compress the gradients, some reducing the per-iteration communication complexity to $\mathcal{O}(k)$, where $k\ll n$. In this paper, we introduce a strategy called two-level gradient averaging (A2SGD) to consolidate all gradients down to merely two local averages per worker before the computation of two global averages for an updated model. A2SGD also retains local errors to maintain the variance for fast convergence. Our analysis shows that A2SGD converges similar to the default distributed SGD algorithm. Our evaluation validates the conclusion and demonstrates that A2SGD significantly reduces the communication traffic per worker, and improves the overall training time of LSTM-PTB by $3.2\times$ and $23.2\times$, compared to Top-K and QSGD, respectively. We evaluate the effectiveness of our approach using two kinds of optimizers, SGD and Adam. Also, our evaluation with various communication options demonstrates the strength of our approach both in terms of communication reduction and convergence. To the best of our knowledge, A2SGD is the first to achieve $\mathcal{O}$ (1) communication complexity per worker without incurring a significant accuracy degradation of DNN models while communicating only two scalars representing gradients per worker for distributed SGD.