On Large-Cohort Training for Federated Learning
On Large-Cohort Training for Federated Learning
复制标题
DOI：
--
复制
发表时间：
2021-06
期刊：
影响因子：
--
通讯作者：
Zachary B. Charles;Zachary Garrett;Zhouyuan Huo;Sergei Shmulyian;Virginia Smith
中科院分区：
文献类型：
--
作者：
Zachary B. Charles;Zachary Garrett;Zhouyuan Huo;Sergei Shmulyian;Virginia Smith
联合学习方法通常通过迭代采样来自一组客户的更新来学习模型。在这项工作中，我们探索了每一轮抽样的客户数量(队列大小)如何影响学习模型的质量和联合学习算法的训练动态。我们的工作提出了三个基本问题。首先，当尝试将联合学习扩展到更大的队列时，会出现哪些挑战？第二，联合学习中的队列大小与集中学习中的批次大小有什么相似之处？最后，我们如何设计有效利用更大队列规模的联合学习方法？我们在广泛的实证评估的基础上给出了这些问题的部分答案。我们的工作突出了由于使用更大的队列而产生的一些挑战。虽然其中一些问题(如泛化问题和回报递减)类似于大批量培训挑战，但另一些问题(包括培训失败和公平问题)是联合学习所特有的。
Federated learning methods typically learn a model by iteratively sampling updates from a population of clients. In this work, we explore how the number of clients sampled at each round (the cohort size) impacts the quality of the learned model and the training dynamics of federated learning algorithms. Our work poses three fundamental questions. First, what challenges arise when trying to scale federated learning to larger cohorts? Second, what parallels exist between cohort sizes in federated learning and batch sizes in centralized learning? Last, how can we design federated learning methods that effectively utilize larger cohort sizes? We give partial answers to these questions based on extensive empirical evaluation. Our work highlights a number of challenges stemming from the use of larger cohorts. While some of these (such as generalization issues and diminishing returns) are analogs of large-batch training challenges, others (including training failures and fairness concerns) are unique to federated learning.