Communication-Adaptive Stochastic Gradient Methods for Distributed Learning
Communication-Adaptive Stochastic Gradient Methods for Distributed Learning
复制标题
DOI：
10.1109/tsp.2021.3099977
复制
发表时间：
2021-01-01
影响因子：
5.4
通讯作者：
Yin, Wotao
中科院分区：
文献类型：
--
作者：
Chen, Tianyi;Sun, Yuejiao;Yin, Wotao
本文的目标是开发以有效通信的方式解决分布式学习问题的算法，通过推广最近的惰性聚合梯度（LAG）方法来处理随机梯度 - 证明新方法 LASG 的名称是正确的。虽然 LAG 可以有效地减少通信而不牺牲收敛速度，但我们表明它仅适用于确定性梯度。我们为 LASG 引入了专为随机梯度定制的新规则和分析，因此它可以有效地节省分布式随机梯度下降的下载、上传或两者。 LASG 取得了令人印象深刻的实证性能 - 它通常可以将总通信量节省一个数量级。 LASG可以与梯度量化一起使用，带来更多的节省。
This paper targets developing algorithms for solving distributed learning problems in a communication-efficient fashion, by generalizing the recent method of lazily aggregated gradient (LAG) to deal with stochastic gradient - justifying the name of the new method LASG. While LAG is effective at reducing communication without sacrificing the rate of convergence, we show it only works with deterministic gradients. We introduce new rules and analysis for LASG that are tailored for stochastic gradients, so it effectively saves downloads, uploads, or both for distributed stochastic gradient descent. LASG achieves impressive empirical performance - it typically saves total communication by an order of magnitude. LASG can be used together with gradient quantization to bring more savings.