Communication-Efficient and Model-Heterogeneous Personalized Federated Learning via Clustered Knowledge Transfer
Communication-Efficient and Model-Heterogeneous Personalized Federated Learning via Clustered Knowledge Transfer
复制标题
DOI：
10.1109/jstsp.2022.3231527
复制
发表时间：
2023-01-01
影响因子：
7.5
通讯作者：
Joshi, Gauri
中科院分区：
文献类型：
--
作者：
Cho, Yae Jee;Wang, Jianyu;Joshi, Gauri
个性化联合学习（PFL）旨在训练能够在各个边缘设备的数据上表现良好的模型，其中边缘设备（客户端）通常是物联网设备，例如我们的手机。跨设备设置的参与客户端一般具有异构系统能力和有限的通信带宽。然而，PFL 最近的许多工作都忽视了边缘设备的这种实用特性，这些工作在所有客户端上使用相同的模型架构，并通过直接通信模型参数而产生高昂的通信成本。在我们的工作中，我们提出了一种新颖实用的 PFL 框架，名为 COMET，客户可以使用自己选择的异构模型，并且不直接将模型参数传达给其他方。相反，COMET 使用集群共蒸馏，其中客户使用知识蒸馏将其知识转移到具有相似数据分布的其他客户。这为边缘设备提供了一个实用的 PFL 框架，通过减轻大型模型通信的繁重通信负担，通过物联网网络进行训练。我们从理论上展示了 COMET 的收敛性和泛化特性，并从经验上表明，与其他最先进的 PFL 方法相比，COMET 实现了较高的测试精度，并且通信成本降低了几个数量级，同时允许客户端模型异构性。
Personalized federated learning (PFL) aims to train model(s) that can perform well on the individual edge-devices' data where the edge-devices (clients) are usually IoT devices like our mobile phones. The participating clients for cross-device settings, in general, have heterogeneous system capabilities and limited communication bandwidth. Such practical properties of the edge-devices, however, are overlooked by many recent work in PFL, which use the same model architecture across all clients and incur high communication cost by directly communicating the model parameters. In our work, we propose a novel and practical PFL framework named COMET where clients can use heterogeneous models of their own choice and do not directly communicate their model parameters to other parties. Instead, COMET uses clustered codistillation, where clients use knowledge distillation to transfer their knowledge to other clients with similar data distributions. This presents a practical PFL framework for the edge-devices to train through IoT networks by lifting the heavy communication burden of communicating large models. We theoretically show the convergence and generalization properties of COMET and empirically show that COMET achieves high test accuracy with several orders of magnitude lower communication cost while allowing client model heterogeneity compared to the other state-of-the-art PFL methods.