Communication-efficient Subspace Methods for High-dimensional Federated Learning
Communication-efficient Subspace Methods for High-dimensional Federated Learning
复制标题
DOI：
10.1109/msn53354.2021.00085
复制
发表时间：
2021-12
期刊：
影响因子：
--
通讯作者：
Zai Shi;A. Eryilmaz
中科院分区：
文献类型：
--
作者：
Zai Shi;A. Eryilmaz
作为一种在边缘计算基础设施中使用机器学习过程的新兴技术，联邦学习（FL）引起了工业界和学术界的极大兴趣。在本文中，我们考虑了FL在无线设置中的潜在挑战，即从边缘设备到中央服务器的上行链路通信容量有限。这对于FL中的机器学习任务（例如训练深度神经网络）尤其重要，这些任务具有极高的维度，可能会大大增加通信负担。为了解决这个问题，我们首先提出了一个基本的方法，称为子空间随机梯度下降联邦学习（FL-SSGD）引入子空间方法的思想。通过理论分析，我们表明，通过选择适当的子空间矩阵FL-SSGD，我们可以减少上行链路的通信成本相比，经典的FedAvg方法。为了改进FL-SSGD，我们提出了另一种称为子空间随机方差降低梯度联邦学习（FL-SSVRG）的方法，该方法具有更快的收敛速度，对目标函数的假设更少。通过在两个FL设置中进行非凸机器学习问题的实验，我们证明了我们的方法与其他通信高效方法相比的优势。
As an emerging technique to employ machine learning processes within an edge computing infrastructure, federated learning (FL) has aroused great interests in both industry and academia. In this paper, we consider a potential challenge of FL in a wireless setup, whereby uplink communication from edge devices to the central server has limited capacity. This is particularly important for machine learning tasks (such as training deep neural networks) in FL with extremely high-dimensional domains that can substantially increase the communication burden. To tackle this challenge, we first propose a basic method called Subspace Stochastic Gradient Descent for Federated Learning (FL-SSGD) to introduce the idea of subspace methods. Through theoretical analysis, we show that by choosing appropriate subspace matrices in FL-SSGD, we can reduce uplink communication costs compared to classical FedAvg method. To improve FL-SSGD, we then propose another method called Subspace Stochastic Variance Reduced Gradient for Federated Learning (FL-SSVRG) that has a faster convergence rate with less assumptions on objective functions. By conducting experiments of a nonconvex machine learning problem in two FL setups, we demonstrate the advantages of our methods compared to other communication-efficient methods.