FedAT: A Communication-Efficient Federated Learning Method with Asynchronous Tiers under Non-IID Data
FedAT: A Communication-Efficient Federated Learning Method with Asynchronous Tiers under Non-IID Data
复制标题
DOI：
--
复制
发表时间：
2020-10
期刊：
影响因子：
--
通讯作者：
Zheng Chai;Yujing Chen;Liang Zhao;Yue Cheng;H. Rangwala
中科院分区：
文献类型：
--
作者：
Zheng Chai;Yujing Chen;Liang Zhao;Yue Cheng;H. Rangwala
联邦学习 (FL) 涉及在大规模分布式设备上训练模型，同时保持训练数据本地化。这种形式的协作学习暴露了模型收敛速度、模型准确性、客户端之间的平衡和通信成本之间的新权衡，带来了新的挑战，包括：(1) 落后者问题，即客户端由于数据或（计算和网络）资源异构性而滞后；(2) 通信瓶颈，即大量客户端将其本地更新传送到中央服务器，并使服务器成为瓶颈。许多现有的 FL 方法专注于仅沿着权衡空间的一维进行优化。现有的解决方案使用异步模型更新或基于分层的同步机制来解决掉队问题。然而，异步方法很容易造成网络通信瓶颈，而分层可能会引入偏差，因为分层有利于更快的层和更短的响应延迟。为了解决这些问题，我们提出了 FedAT，这是一种在非独立同分布下具有异步层的新型联合学习方法。数据。 FedAT 协同结合了同步层内训练和异步跨层训练。通过分层桥接同步和异步训练，FedAT 最大限度地减少了落后者效应，提高了收敛速度和测试精度。 FedAT 使用落后者感知的加权聚合启发式方法来引导和平衡训练，以进一步提高准确性。 FedAT 使用高效、基于折线编码的压缩算法来压缩上行链路和下行链路通信，从而最大限度地降低通信成本。结果表明，与最先进的 FL 方法相比，FedAT 将预测性能提高了 21.09%，并将通信成本降低了 8.5 倍。
Federated learning (FL) involves training a model over massive distributed devices, while keeping the training data localized. This form of collaborative learning exposes new tradeoffs among model convergence speed, model accuracy, balance across clients, and communication cost, with new challenges including: (1) straggler problem, where the clients lag due to data or (computing and network) resource heterogeneity, and (2) communication bottleneck, where a large number of clients communicate their local updates to a central server and bottleneck the server. Many existing FL methods focus on optimizing along only one dimension of the tradeoff space. Existing solutions use asynchronous model updating or tiering-based synchronous mechanisms to tackle the straggler problem. However, the asynchronous methods can easily create a network communication bottleneck, while tiering may introduce biases as tiering favors faster tiers with shorter response latencies. To address these issues, we present FedAT, a novel Federated learning method with Asynchronous Tiers under Non-i.i.d. data. FedAT synergistically combines synchronous intra-tier training and asynchronous cross-tier training. By bridging the synchronous and asynchronous training through tiering, FedAT minimizes the straggler effect with improved convergence speed and test accuracy. FedAT uses a straggler-aware, weighted aggregation heuristic to steer and balance the training for further accuracy improvement. FedAT compresses the uplink and downlink communications using an efficient, polyline-encoding-based compression algorithm, therefore minimizing the communication cost. Results show that FedAT improves the prediction performance by up to 21.09%, and reduces the communication cost by up to 8.5x, compared to state-of-the-art FL methods.