Faster Federated Learning With Decaying Number of Local SGD Steps
Faster Federated Learning With Decaying Number of Local SGD Steps
复制标题
更快的联邦学习，减少本地SGD步骤数
DOI：
10.1109/tpds.2023.3277367
复制
发表时间：
2023-05
影响因子：
5.3
通讯作者：
Jed Mills;Jia Hu;Geyong Min
中科院分区：
文献类型：
--
作者：
Jed Mills;Jia Hu;Geyong Min
在联邦学习（FL）中，通过互联网连接的客户端设备协同训练机器学习模型，而无需与中央服务器或其他客户端共享其私有数据。开创性的联邦平均（FedAvg）算法通过在客户端执行几轮本地训练，然后进行模型平均，来训练单个全局模型。fedag可以通过在每一轮对客户端执行更多的随机梯度下降（SGD）步骤来提高训练的沟通效率。然而，真实FL中的客户端数据是高度异构的，当每轮在客户端上执行$K>1 $K>1步骤的SGD时，这已被广泛证明会减缓模型收敛并损害最终性能。在本文中，我们提出在训练过程中衰减$K$K，与使用固定$K$K相比，这可以共同提高FL模型的最终性能，同时减少挂钟时间和训练的总计算成本。我们分析了强凸目标下具有衰减$K$K的fedag的收敛性，提供了对收敛性质的新见解，并推导了$K$K的三个理论驱动的衰减计划。然后，我们在四个基准FL数据集（FEMNIST, CIFAR100, Sentiment140, Shakespeare）上进行了彻底的实验，以显示我们的方法在现实世界的收敛时间，计算成本和泛化性能方面的实际好处。
In Federated Learning (FL) client devices connected over the internet collaboratively train a machine learning model without sharing their private data with a central server or with other clients. The seminal Federated Averaging (FedAvg) algorithm trains a single global model by performing rounds of local training on clients followed by model averaging. FedAvg can improve the communication-efficiency of training by performing more steps of Stochastic Gradient Descent (SGD) on clients in each round. However, client data in real-world FL is highly heterogeneous, which has been extensively shown to slow model convergence and harm final performance when $K > 1$K>1 steps of SGD are performed on clients per round. In this article we propose decaying $K$K as training progresses, which can jointly improve the final performance of the FL model whilst reducing the wall-clock time and the total computational cost of training compared to using a fixed $K$K. We analyse the convergence of FedAvg with decaying $K$K for strongly-convex objectives, providing novel insights into the convergence properties, and derive three theoretically-motivated decay schedules for $K$K. We then perform thorough experiments on four benchmark FL datasets (FEMNIST, CIFAR100, Sentiment140, Shakespeare) to show the real-world benefit of our approaches in terms of real-world convergence time, computational cost, and generalisation performance.