FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Models
FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Models
复制标题
DOI：
10.1109/icdcs54860.2022.00094
复制
发表时间：
2021-09
期刊：
影响因子：
--
通讯作者：
Lan Zhang;Dapeng Wu;Xiaoyong Yuan
中科院分区：
文献类型：
--
作者：
Lan Zhang;Dapeng Wu;Xiaoyong Yuan
联合学习使多个分布式设备能够协作学习共享的预测模型，而无需集中其设备上的数据。大多数当前的算法需要具有相同结构和大小的设备上模型的本地训练的可比的个人努力，然而，这阻碍了资源受限的设备的参与。鉴于当今广泛而异构的设备，在本文中，我们提出了一个创新的联邦学习框架，通过零射击知识转移，由FedZKT命名的异构设备上的模型。具体来说，FedZKT允许设备根据其本地资源独立确定设备上的模型。为了在这些异构的设备上模型之间实现知识转移，设计了一种零触发蒸馏方法，而不需要任何私有设备上数据的先决条件，这与基于公共数据集或预先训练的数据生成器的某些先前研究相反。此外，这个计算密集型提取任务被分配给服务器，以允许资源受限的设备参与，其中生成器通过收集的设备上模型的集合进行对抗学习。然后，将提取的中心知识以相应的设备上模型参数的形式发送回来，这些参数可以很容易地在设备侧被吸收。大量的实验研究证明了FedZKT对设备上知识不可知、设备上模型异构和其他具有挑战性的联邦学习场景（如异构设备上数据和落伍者效应）的有效性和鲁棒性。
Federated learning enables multiple distributed devices to collaboratively learn a shared prediction model without centralizing their on-device data. Most of the current algorithms require comparable individual efforts for local training with the same structure and size of on-device models, which, however, impedes participation from resource-constrained devices. Given the widespread yet heterogeneous devices nowadays, in this paper, we propose an innovative federated learning framework with heterogeneous on-device models through Zero-shot Knowledge Transfer, named by FedZKT. Specifically, FedZKT allows devices to independently determine the on-device models upon their local resources. To achieve knowledge transfer across these heterogeneous on-device models, a zero-shot distillation approach is designed without any prerequisites for private on-device data, which is contrary to certain prior research based on a public dataset or a pre-trained data generator. Moreover, this compute-intensive distillation task is assigned to the server to allow the participation of resource-constrained devices, where a generator is adversarially learned with the ensemble of collected on-device models. The distilled central knowledge is then sent back in the form of the corresponding on-device model parameters, which can be easily absorbed on the device side. Extensive experimental studies demonstrate the effectiveness and robustness of FedZKT towards on-device knowledge agnostic, on-device model heterogeneity, and other challenging federated learning scenarios, such as heterogeneous on-device data and straggler effects.