Data-Free Knowledge Distillation for Heterogeneous Federated Learning
Data-Free Knowledge Distillation for Heterogeneous Federated Learning
复制标题
DOI：
--
复制
发表时间：
2021-05
期刊：
影响因子：
--
通讯作者：
Zhuangdi Zhu;Junyuan Hong;Jiayu Zhou
中科院分区：
文献类型：
--
作者：
Zhuangdi Zhu;Junyuan Hong;Jiayu Zhou
联邦学习（FL）是一种分散的机器学习范式，其中全局服务器迭代地聚合本地用户的模型参数，而无需访问他们的数据。用户的异质性给FL带来了巨大的挑战，这可能会导致全局模型漂移，收敛缓慢。最近出现了知识蒸馏来解决这个问题，通过使用来自异构用户的聚合知识来细化服务器模型，而不是直接聚合他们的模型参数。然而，这种方法依赖于代理数据集，使得它不切实际，除非满足这样的先决条件。此外，集成知识没有被充分利用来指导局部模型学习，这可能反过来影响聚合模型的质量。在这项工作中，我们提出了一种无数据的知识蒸馏方法来解决异构FL，其中服务器学习轻量级生成器以无数据的方式集成用户信息，然后将其广播给用户，使用学习到的知识作为归纳偏差来调节本地训练。基于理论的实证研究表明，与现有技术相比，我们的方法使用更少的通信轮数，以更好的泛化性能促进FL。
Federated Learning (FL) is a decentralized machine-learning paradigm in which a global server iteratively aggregates the model parameters of local users without accessing their data. User heterogeneity has imposed significant challenges to FL, which can incur drifted global models that are slow to converge. Knowledge Distillation has recently emerged to tackle this issue, by refining the server model using aggregated knowledge from heterogeneous users, other than directly aggregating their model parameters. This approach, however, depends on a proxy dataset, making it impractical unless such prerequisite is satisfied. Moreover, the ensemble knowledge is not fully utilized to guide local model learning, which may in turn affect the quality of the aggregated model. In this work, we propose a data-free knowledge distillation approach to address heterogeneous FL, where the server learns a lightweight generator to ensemble user information in a data-free manner, which is then broadcasted to users, regulating local training using the learned knowledge as an inductive bias. Empirical studies powered by theoretical implications show that, our approach facilitates FL with better generalization performance using fewer communication rounds, compared with the state-of-the-art.