Quality-Aware Distributed Computation for Cost-Effective Non-Convex and Asynchronous Wireless Federated Learning
Quality-Aware Distributed Computation for Cost-Effective Non-Convex and Asynchronous Wireless Federated Learning
复制标题
DOI：
10.23919/wiopt52861.2021.9589660
复制
发表时间：
2021-10
期刊：
影响因子：
--
通讯作者：
Yuxi Zhao;Xiaowen Gong
中科院分区：
文献类型：
--
作者：
Yuxi Zhao;Xiaowen Gong
无线联合学习（WFL）以分布式方式在无线边缘设备上训练机器学习（ML）模型，而无需从用户那里收集数据。在WFL中，局部模型更新的质量取决于局部随机梯度的方差，由用于计算更新的小批量数据大小确定。在本文中，我们研究了WFL的质量感知分布式计算与非凸问题和异步算法，使用小批量大小作为一个“旋钮”来控制用户的本地更新的质量。我们首先将训练损失的性能界限表征为训练过程中局部更新质量的函数，对于非凸和异步设置。我们的研究结果表明，局部更新的质量对训练损失的影响1）随着用于非凸学习的局部更新的步长而增加，以及2）当有更多其他用户的局部更新与异步学习的局部更新（取决于更新延迟）相结合时增加。基于这些有用的见解，我们设计了信道感知的自适应算法，该算法基于本地更新的质量对训练损失以及用户的无线信道条件（决定更新延迟）和计算成本的影响来确定用户在训练过程中的小批量大小。我们使用模拟来评估所提出的品质感知自适应演算法，其显示出改善的学习准确度与学习成本。
Wireless federated learning (WFL) trains machine learning (ML) models on wireless edge devices in a distributed manner without the need of collecting data from users. In WFL, the quality of a local model update depends on the variance of the local stochastic gradient, determined by the mini-batch data size used to compute the update. In this paper, we study quality-aware distributed computation for WFL with non-convex problems and asynchronous algorithms, using mini-batch size as a "knob" to control the quality of users' local updates. We first characterize performance bounds on the training loss as a function of local updates' quality over the training process, for both non-convex and asynchronous settings. Our findings reveal that the impact of a local update's quality on the training loss 1) increases with the stepsize used for that local update for non- convex learning, and 2) increases when there are more other users' local updates which are coupled with that local update (depending on the update delays) for asynchronous learning. Based on these useful insights, we design channel-aware adaptive algorithms that determine users' mini-batch sizes over the training process, based on the impacts of local updates' quality on the training loss as well as users' wireless channel conditions (which determine the update delays) and computation costs. We evaluate the proposed quality- aware adaptive algorithms using simulations, which demonstrate improved learning accuracy and learning cost.