Adaptive Model Pruning for Communication and Computation Efficient Wireless Federated Learning
Adaptive Model Pruning for Communication and Computation Efficient Wireless Federated Learning
复制标题
DOI：
10.1109/twc.2023.3342626
复制
发表时间：
2023
影响因子：
10.4
通讯作者：
Zhixiong Chen;Wenqiang Yi;Hyundong Shin;Arumgam Nallanathan
中科院分区：
文献类型：
--
作者：
Zhixiong Chen;Wenqiang Yi;Hyundong Shin;Arumgam Nallanathan
-大多数现有的无线联合学习(FL)研究都集中在同类模型设置上，其中设备训练相同的本地模型。在这种情况下，通信和计算能力较差的设备可能会延迟全局模型的更新，并降低FL的性能。此外，在同质模型设置中，全局模型的比例受到具有最低能力的设备的限制。为了应对这些挑战，提出了一种基于自适应模型修剪的FL(AMP-FL)框架，边缘服务器通过修剪全局模型来动态生成子模型，以适应设备的局部训练，以适应设备的异构性计算能力和时变的信道条件。由于设备子模型的不同结构参与全局模型更新可能会对训练收敛产生负面影响，我们提出用设备的历史梯度来补偿剪枝模型区域的梯度。然后，我们引入了信息年龄(AOI)度量来刻画局部梯度的陈旧性，并从理论上分析了AMP-FL的收敛行为。收敛界建议对梯度AOI较大的设备进行调度，对AOI较小的设备进行模型区域的剪枝，以提高学习性能。受此启发，我们设计了一个新的目标函数，即局部梯度的平均AOI，将不明确的全局损失最小化问题转化为设备调度、模型剪枝和资源块(RB)分配设计的易处理问题。通过详细分析，推导出最优模型剪枝策略，将RB分配问题转化为可有效求解的等价线性规划问题。实验结果证明了该方法的有效性和优越性。在MNIST和CIFAR-10数据集上，与同质模型设置的FL方案相比，提出的AMP-FL方案能够获得1.9倍和1.6倍的FL加速。
—Most existing wireless federated learning (FL) studies focused on homogeneous model settings where devices train identical local models. In this setting, the devices with poor communication and computation capabilities may delay the global model update and degrade the performance of FL. Moreover, in the homogenous model settings, the scale of the global model is restricted by the device with the lowest capability. To tackle these challenges, this work proposes an adaptive model pruning-based FL (AMP-FL) framework, where the edge server dynamically generates sub-models by pruning the global model for devices’ local training to adapt their heterogeneous computation capabilities and time-varying channel conditions. Since the involvement of diverse structures of devices’ sub-models in the global model updating may negatively affect the training convergence, we propose compensating for the gradients of pruned model regions by devices’ historical gradients. We then introduce an age of information (AoI) metric to characterize the staleness of local gradients and theoretically analyze the convergence behaviour of AMP-FL. The convergence bound suggests scheduling devices with large AoI of gradients and pruning the model regions with small AoI for devices to improve the learning performance. Inspired by this, we define a new objective function, i.e., the average AoI of local gradients, to transform the inexplicit global loss minimization problem into a tractable one for device scheduling, model pruning, and resource block (RB) allocation design. Through detailed analysis, we derive the optimal model pruning strategy and transform the RB allocation problem into equivalent linear programming that can be effectively solved. Experimental results demonstrate the effectiveness and superiority of the proposed approaches. The proposed AMP-FL is capable of achieving 1.9x and 1.6x speed up for FL on MNIST and CIFAR-10 datasets in comparison with the FL schemes with homogeneous model settings.