A Novel Framework for the Analysis and Design of Heterogeneous Federated Learning
A Novel Framework for the Analysis and Design of Heterogeneous Federated Learning
复制标题
DOI：
10.1109/tsp.2021.3106104
复制
发表时间：
2021-01-01
影响因子：
5.4
通讯作者：
Poor, H. Vincent
中科院分区：
文献类型：
--
作者：
Wang, Jianyu;Liu, Qinghua;Poor, H. Vincent
在联邦学习中，客户端本地数据集和计算速度的异质性导致每个客户端在每个通信回合中执行的本地更新数量存在很大差异。这种模型的朴素加权聚合导致目标不一致，即全局模型收敛到不匹配目标函数的稳定点，该目标函数可以与真实目标任意不同。本文提供了一个通用的框架来分析联邦优化算法的收敛性与异构的本地训练过程在客户端。分析是在光滑非凸和强凸的情况下进行的，也可以扩展到部分客户参与的情况。此外，它包含了以前提出的方法，如FedAvg和FedProx，并提供了第一个原则性的理解解决方案的偏见和收敛速度放缓，由于客观的不一致性。利用这种分析的见解，我们提出了FedNova，一种归一化的平均方法，消除了客观的不一致性，同时保持快速的误差收敛。
In federated learning, heterogeneity in the clients' local datasets and computation speeds results in large variations in the number of local updates performed by each client in each communication round. Naive weighted aggregation of such models causes objective inconsistency, that is, the global model converges to a stationary point of a mismatched objective function which can be arbitrarily different from the true objective. This paper provides a general framework to analyze the convergence of federated optimization algorithms with heterogeneous local training progress at clients. The analyses are conducted for both smooth non-convex and strongly convex settings, and can also be extended to partial client participation case. Additionally, it subsumes previously proposed methods such as FedAvg and FedProx, and provides the first principled understanding of the solution bias and the convergence slowdown due to objective inconsistency. Using insights from this analysis, we propose FedNova, a normalized averaging method that eliminates objective inconsistency while preserving fast error convergence.