An Efficient Framework for Clustered Federated Learning
An Efficient Framework for Clustered Federated Learning
复制标题
DOI：
10.1109/tit.2022.3192506
复制
发表时间：
2022-12-01
影响因子：
2.5
通讯作者：
Ramchandran, Kannan
中科院分区：
文献类型：
--
作者：
Ghosh, Avishek;Chung, Jichan;Ramchandran, Kannan
我们解决了联邦学习（FL）的问题，其中用户被分布并划分到集群中。这种设置捕获了这样的设置：不同的用户组有自己的目标（学习任务），但通过将他们的数据与同一集群中的其他用户（相同的学习任务）聚集在一起，他们可以利用数量上的优势来执行更有效的联合学习。对于这种新的聚类联邦学习框架，我们提出了迭代联邦聚类算法（IFCA），该算法交替估计用户的聚类身份并通过梯度下降优化用户聚类的模型参数。首先分析了该算法在具有平方损失的线性模型下的收敛速度，然后分析了一般强凸函数和光滑函数的收敛速度。我们证明了在这两种情况下，良好的初始化，IFCA保证收敛，并讨论了统计错误率的最优性。特别是对于有两个聚类的线性模型，只要初始化比随机略好，我们就可以保证算法收敛。当聚类结构不明确时，我们提出将IFCA与多任务学习中的权值共享技术相结合来训练模型。实验表明，通过随机初始化和多次重启来放宽初始化要求，该算法仍然可以成功。实验结果表明，我们的算法在神经网络等非凸问题上是有效的。我们在几个集群的FL基准测试中展示了IFCA相对于基线的优势。
We address the problem of federated learning (FL) where users are distributed and partitioned into clusters. This setup captures settings where different groups of users have their own objectives (learning tasks) but by aggregating their data with others in the same cluster (same learning task), they can leverage the strength in numbers in order to perform more efficient federated learning. For this new framework of clustered federated learning, we propose the Iterative Federated Clustering Algorithm (IFCA), which alternately estimates the cluster identities of the users and optimizes model parameters for the user clusters via gradient descent. We analyze the convergence rate of this algorithm first in a linear model with squared loss and then for generic strongly convex and smooth loss functions. We show that in both settings, with good initialization, IFCA is guaranteed to converge, and discuss the optimality of the statistical error rate. In particular, for the linear model with two clusters, we can guarantee that our algorithm converges as long as the initialization is slightly better than random. When the clustering structure is ambiguous, we propose to train the models by combining IFCA with the weight sharing technique in multi-task learning. In the experiments, we show that our algorithm can succeed even if we relax the requirements on initialization with random initialization and multiple restarts. We also present experimental results showing that our algorithm is efficient in non-convex problems such as neural networks. We demonstrate the benefits of IFCA over the baselines on several clustered FL benchmarks.