FetchSGD: Communication-Efficient Federated Learning with Sketching
FetchSGD: Communication-Efficient Federated Learning with Sketching
复制标题
DOI：
--
复制
发表时间：
2020-07
期刊：
影响因子：
--
通讯作者：
D. Rothchild;Ashwinee Panda;Enayat Ullah;Nikita Ivkin;I. Stoica;Vladimir Braverman;Joseph Gonzalez-Joseph-Gonzale
中科院分区：
文献类型：
--
作者：
D. Rothchild;Ashwinee Panda;Enayat Ullah;Nikita Ivkin;I. Stoica;Vladimir Braverman;Joseph Gonzalez-Joseph-Gonzale
现有的联邦学习方法存在通信瓶颈以及由于客户端参与稀少而导致的收敛问题。在本文中，我们介绍了一种称为 FetchSGD 的新颖算法来克服这些挑战。 FetchSGD 使用计数草图压缩模型更新，然后利用草图的可合并性来组合来自许多工作人员的模型更新。 FetchSGD 设计的一个关键见解是，由于计数草图是线性的，因此动量和误差累积都可以在草图中进行。这使得算法能够将动量和误差累积从客户端转移到中央聚合器，克服客户端参与稀疏的挑战，同时仍然实现高压缩率和良好的收敛性。我们证明了 FetchSGD 具有良好的收敛保证，并通过训练两个残差网络和一个 Transformer 模型证明了其实证有效性。
Existing approaches to federated learning suffer from a communication bottleneck as well as convergence issues due to sparse client participation. In this paper we introduce a novel algorithm, called FetchSGD, to overcome these challenges. FetchSGD compresses model updates using a Count Sketch, and then takes advantage of the mergeability of sketches to combine model updates from many workers. A key insight in the design of FetchSGD is that, because the Count Sketch is linear, momentum and error accumulation can both be carried out within the sketch. This allows the algorithm to move momentum and error accumulation from clients to the central aggregator, overcoming the challenges of sparse client participation while still achieving high compression rates and good convergence. We prove that FetchSGD has favorable convergence guarantees, and we demonstrate its empirical effectiveness by training two residual networks and a transformer model.