BS-pFL: Enabling Low-Cost Personalized Federated Learning by Exploring Weight Gradient Sparsity
BS-pFL: Enabling Low-Cost Personalized Federated Learning by Exploring Weight Gradient Sparsity
复制标题
DOI：
10.1109/ijcnn55064.2022.9892137
复制
发表时间：
2022-07
期刊：
影响因子：
--
通讯作者：
Lening Wang;Manojna Sistla;Mingsong Chen;Xin Fu
中科院分区：
文献类型：
--
作者：
Lening Wang;Manojna Sistla;Mingsong Chen;Xin Fu
卷积神经网络(CNN)的最新进展已经在许多应用中取得了惊人的成功。CNN的破纪录性能通常是以高昂的训练成本为代价的，因此所有的训练数据通常都是在强大的集中服务器端处理的，这引发了隐私问题。联合学习(FL)是一种基于移动设备的分布式机器学习方法，用于训练全局模型，同时将分散的数据保存在设备上以保护数据隐私。然而，在移动客户端上部署FL有两个主要限制。首先，在客户端，移动设备上有限的通信和计算资源不能很好地支持完整的训练迭代。其次，在服务器端，传统FL只聚合所有客户端的公共输出，而没有将模型个性化到每个客户端，这是当客户端具有异质数据分布时缺少的重要特征。在这项工作中，我们的目标是通过关注FL中最重要的交换参数-权重梯度来实现低成本的个性化FL，同时控制计算和通信开销。我们首先观察到客户端计算的权重梯度具有高度稀疏性，并且权重梯度中的稀疏模式可以通过对比特序列(称为比特流)的非常简单的逐比特操作来预测，而不是进行昂贵的高精度计算来确定它们。此外，根据每个客户端的本地训练数据的分布，在每个客户端的上传的权重梯度中显示出唯一的模式。在这种模式的指导下，每个客户都可以获得一个个性化的聚合模型来适应自己的数据。因此，我们利用比特流来预测权重梯度稀疏性，以便在每个设备上进行低成本的训练，同时，比特流被用来表示每个客户端唯一的权重梯度稀疏模式，这将指导模型个性化。实验结果表明，该框架在保证个性化精度的前提下，计算效率平均提高3.5倍(最高可达4.2倍)，通信开销平均降低23%(最高可达41%)。
Recent advancements in Convolution Neural Networks (CNNs) have achieved amazing success in numerous applications. The record-breaking performance of CNNs is usually at the prohibitive training costs, thus all training data are usually processed at the powerful centralized server side, which rises privacy concerns. Federated learning (FL) is a distributed machine learning method over mobile devices to train a global model while keeping decentralized data on devices to preserve the data privacy. However, there are two major limitations to deploy FL on mobile clients. Firstly, on the client side, the limited communication and computation resources on mobile devices cannot well support the full training iterations. Secondly, on the server side, conventional FL only aggregate a common output for all the clients without personalizing the model to each client, which is an important missing feature when clients have heterogeneous data distributions. In this work, we aim to enable low-cost personalized FL by focusing on the weight gradients which are the most important exchanging parameters in FL and meanwhile, dominating the computation and communication cost. We first observe that the client's calculated weight gradients have high sparsity, and the sparse pattern in weight gradients could be predicted via very simple bit-wise operations on a sequence of bits (named bit-stream) instead of conducting expensive high-precision calculations to determine them. Furthermore, a unique pattern is exhibited in each client's uploaded weight gradients according to the distribution of its local training data. Guided by this pattern, each client can get a personalized aggregated model to fit its own data. Hence, we leverage bit-streams to predict weight gradients sparsity for low-cost training on each device, and meanwhile, bit-streams are used to represent the unique sparse pattern of the weight gradient for each client which will guide the model personalization. From our experiments, our proposed framework can improve the computation efficiency by 3.5× on average (up to 4.2×) and reduce the communication cost by 23% on average (up to 41%) while still achieving the state-of-the-art personalized accuracy.