LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning
LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning
复制标题
DOI：
--
复制
发表时间：
2018-05
影响因子：
9
通讯作者：
Tianyi Chen;G. Giannakis;Tao Sun;W. Yin
中科院分区：
文献类型：
--
作者：
Tianyi Chen;G. Giannakis;Tao Sun;W. Yin
本文提出了一类新的梯度方法，用于分布式机器学习，自适应地跳过梯度计算，以减少通信和计算来学习。设计简单的规则来检测缓慢变化的梯度，从而触发过时梯度的重用。由此产生的基于梯度的算法被称为懒惰聚合梯度-证明我们的缩写LAG使用。从理论上讲，这种贡献的优点是：i）在强凸，凸和非凸光滑情况下，收敛速度与批量梯度下降相同; ii）如果分布式数据集是异构的（由某些可测量常数量化），由于滞后梯度的自适应重用，实现目标精度所需的通信轮次减少。合成和真实的数据的数值实验证实了一个显着的通信减少相比，替代品。
This paper presents a new class of gradient methods for distributed machine learning that adaptively skip the gradient calculations to learn with reduced communication and computation. Simple rules are designed to detect slowly-varying gradients and, therefore, trigger the reuse of outdated gradients. The resultant gradient-based algorithms are termed Lazily Aggregated Gradient --- justifying our acronym LAG used henceforth. Theoretically, the merits of this contribution are: i) the convergence rate is the same as batch gradient descent in strongly-convex, convex, and nonconvex smooth cases; and, ii) if the distributed datasets are heterogeneous (quantified by certain measurable constants), the communication rounds needed to achieve a targeted accuracy are reduced thanks to the adaptive reuse of lagged gradients. Numerical experiments on both synthetic and real data corroborate a significant communication reduction compared to alternatives.