Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
复制标题
DOI：
--
复制
发表时间：
2020-08
期刊：
影响因子：
--
通讯作者：
Sai Praneeth Karimireddy;Martin Jaggi;Satyen Kale;M. Mohri;Sashank J. Reddi;Sebastian U. Stich;A. Suresh
中科院分区：
文献类型：
--
作者：
Sai Praneeth Karimireddy;Martin Jaggi;Satyen Kale;M. Mohri;Sashank J. Reddi;Sebastian U. Stich;A. Suresh
联邦学习是一个具有挑战性的优化问题，因为不同客户机之间的数据具有异质性。这种异质性已经被观察到会导致客户端漂移，并显著降低为这种设置设计的算法的性能。相比之下，集中收集数据的集中式学习没有这种漂移，并且在动量、适应性等创新方面取得了很大的经验和理论进步。在这项工作中，我们提出了一个通用框架Mime，它可以减轻客户端漂移，并将任意集中式优化算法（例如\ SGD， Adam等）适应联邦学习。Mime在每个客户端更新步骤中使用控制变量和服务器级统计数据（例如动量）的组合，以确保每个本地更新都模仿集中式方法。我们全面的理论和实证分析有力地确立了Mime优于其他基线的优势。
Federated learning is a challenging optimization problem due to the heterogeneity of the data across different clients. Such heterogeneity has been observed to induce client drift and significantly degrade the performance of algorithms designed for this setting. In contrast, centralized learning with centrally collected data does not experience such drift, and has seen great empirical and theoretical progress with innovations such as momentum, adaptivity, etc. In this work, we propose a general framework Mime which mitigates client-drift and adapts arbitrary centralized optimization algorithms (e.g.\ SGD, Adam, etc.) to federated learning. Mime uses a combination of control-variates and server-level statistics (e.g. momentum) at every client-update step to ensure that each local update mimics that of the centralized method. Our thorough theoretical and empirical analyses strongly establish Mime's superiority over other baselines.