FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization
FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization
复制标题
DOI：
--
复制
发表时间：
2019-09
期刊：
影响因子：
--
通讯作者：
Amirhossein Reisizadeh;Aryan Mokhtari;Hamed Hassani;A. Jadbabaie;Ramtin Pedarsani
中科院分区：
文献类型：
--
作者：
Amirhossein Reisizadeh;Aryan Mokhtari;Hamed Hassani;A. Jadbabaie;Ramtin Pedarsani
联合学习是一种分布式框架，根据该框架，模型在一组设备上进行训练，同时保持数据本地化。该框架面临着几个面向系统的挑战，其中包括（i）通信瓶颈，因为大量的设备将其本地更新上传到参数服务器，以及（ii）可扩展性，因为联合网络由数百万个设备组成。由于这些系统挑战以及与数据统计异质性和隐私问题相关的问题，设计一种可证明有效的联邦学习方法非常重要，但它仍然具有挑战性。在本文中，我们提出了FedPAQ，一种具有周期平均和量化的通信高效的联邦学习方法。FedPAQ依赖于三个关键特性：（1）定期平均，其中模型在设备上本地更新，并且仅在服务器上定期平均;（2）部分设备参与，其中只有一小部分设备参与每轮训练;以及（3）量化消息传递，其中边缘节点在上传到参数服务器之前对其更新进行验证。这些特性解决了联邦学习中的通信和可扩展性挑战。我们还表明，FedPAQ实现了强凸和非凸损失函数的接近最优的理论保证，并实证证明了我们的方法提供的通信计算权衡。
Federated learning is a distributed framework according to which a model is trained over a set of devices, while keeping data localized. This framework faces several systems-oriented challenges which include (i) communication bottleneck since a large number of devices upload their local updates to a parameter server, and (ii) scalability as the federated network consists of millions of devices. Due to these systems challenges as well as issues related to statistical heterogeneity of data and privacy concerns, designing a provably efficient federated learning method is of significant importance yet it remains challenging. In this paper, we present FedPAQ, a communication-efficient Federated Learning method with Periodic Averaging and Quantization. FedPAQ relies on three key features: (1) periodic averaging where models are updated locally at devices and only periodically averaged at the server; (2) partial device participation where only a fraction of devices participate in each round of the training; and (3) quantized message-passing where the edge nodes quantize their updates before uploading to the parameter server. These features address the communications and scalability challenges in federated learning. We also show that FedPAQ achieves near-optimal theoretical guarantees for strongly convex and non-convex loss functions and empirically demonstrate the communication-computation tradeoff provided by our method.