Communication-Efficient Distributed SGD With Compressed Sensing
Communication-Efficient Distributed SGD With Compressed Sensing
复制标题
DOI：
10.1109/lcsys.2021.3137859
复制
发表时间：
2021-12
影响因子：
3
通讯作者：
Yujie Tang;V. Ramanathan;Junshan Zhang;N. Li
中科院分区：
文献类型：
--
作者：
Yujie Tang;V. Ramanathan;Junshan Zhang;N. Li
我们考虑了一组连接到中央服务器的边缘设备上的大规模分布式优化，其中服务器和边缘设备之间有限的通信带宽给优化过程带来了严重的瓶颈。受联邦学习最新进展的启发，我们提出了一种分布式随机梯度下降(SGD)型算法，该算法尽可能地利用梯度的稀疏性来减少通信负担。算法的核心是在设备端使用压缩传感技术对局部随机梯度进行压缩；在服务器端，从噪声聚集的压缩局部梯度中恢复全局随机梯度的稀疏近似。对该算法在信道噪声干扰下的收敛性能进行了理论分析，并通过数值实验验证了该算法的有效性。
We consider large scale distributed optimization over a set of edge devices connected to a central server, where the limited communication bandwidth between the server and edge devices imposes a significant bottleneck for the optimization procedure. Inspired by recent advances in federated learning, we propose a distributed stochastic gradient descent (SGD) type algorithm that exploits the sparsity of the gradient, when possible, to reduce communication burden. At the heart of the algorithm is to use compressed sensing techniques for the compression of the local stochastic gradients at the device side; and at the server side, a sparse approximation of the global stochastic gradient is recovered from the noisy aggregated compressed local gradients. We conduct theoretical analysis on the convergence of our algorithm in the presence of noise perturbation incurred by the communication channels, and also conduct numerical experiments to corroborate its effectiveness.