Efficient and Reliable Overlay Networks for Decentralized Federated Learning
Efficient and Reliable Overlay Networks for Decentralized Federated Learning
复制标题
DOI：
10.1137/21m1465081
复制
发表时间：
2021-12
期刊：
影响因子：
--
通讯作者：
Yifan Hua;Kevin Miller;A. Bertozzi;Chen Qian;Bao Wang
中科院分区：
文献类型：
--
作者：
Yifan Hua;Kevin Miller;A. Bertozzi;Chen Qian;Bao Wang
我们提出了基于$d$-正则扩展图的近优覆盖网络，以加速分布式联邦学习(DFL)并提高其泛化能力。在DFL中，大量的客户端通过覆盖网络连接在一起，它们在不共享原始数据的情况下协作解决机器学习问题。我们的覆盖网络设计结合了谱图理论和DFL的理论收敛和泛化界限。因此，我们提出的覆盖网络加速了收敛，改善了泛化，并增强了对DFL中客户故障的稳健性，并提供了理论保证。此外，我们还提出了一种有效的算法来将给定的图转换为实际的覆盖网络，并在潜在的客户故障后保持网络的拓扑。我们用我们提出的网络在各种基准任务上数值验证了DFL的优势，从图像分类到使用数百个客户端的语言建模。
We propose near-optimal overlay networks based on $d$-regular expander graphs to accelerate decentralized federated learning (DFL) and improve its generalization. In DFL a massive number of clients are connected by an overlay network, and they solve machine learning problems collaboratively without sharing raw data. Our overlay network design integrates spectral graph theory and the theoretical convergence and generalization bounds for DFL. As such, our proposed overlay networks accelerate convergence, improve generalization, and enhance robustness to clients failures in DFL with theoretical guarantees. Also, we present an efficient algorithm to convert a given graph to a practical overlay network and maintaining the network topology after potential client failures. We numerically verify the advantages of DFL with our proposed networks on various benchmark tasks, ranging from image classification to language modeling using hundreds of clients.