C U P C A K E : A C OMPRESSION O PTIMIZER FOR S CALABLE C OMMUNICATION -E FFICIENT D ISTRIBUTED T RAINING
C U P C A K E : A C OMPRESSION O PTIMIZER FOR S CALABLE C OMMUNICATION -E FFICIENT D ISTRIBUTED T RAINING
复制标题
DOI：
--
复制
发表时间：
2023
期刊：
影响因子：
--
通讯作者：
Zhuang Wang;X. Wu;Zhaozhuo Xu;T. Ng
中科院分区：
文献类型：
--
作者：
Zhuang Wang;X. Wu;Zhaozhuo Xu;T. Ng
数据并行分布式训练 (DDT) 是在多个 GPU 上加速深度学习的实际方法。在DDT中，梯度同步的通信是主要的效率瓶颈。人们已经提出了许多梯度压缩（GC）算法来通过减少通信数据量来解决这一通信瓶颈。不幸的是，据观察，GC 在 DDT 中仅实现了适度的性能提升，甚至损害了性能。在本文中，我们认为当前以分层方式部署 GC 的方式减少了通信时间，但代价是不可忽略的压缩开销。为了解决这个问题，我们提出了Cupcake，一种压缩优化器，以充分发挥GC算法在加速DDT方面的优势。它以融合方式应用 GC 算法，并确定可证明的最佳融合策略，以最大限度地提高启用压缩的 DDT 作业的训练吞吐量。实验评估表明，与不使用 GC 的训练相比，使用 Cupcake 的 GC 算法可以在训练吞吐量上实现高达 2.03 倍的加速，并且比以分层方式将 GC 应用于 DDT 的最先进方法加速高达 1.79 倍。
Data-parallel distributed training (DDT) is the de facto way to accelerate deep learning on multiple GPUs. In DDT, communication for gradient synchronization is the major efficiency bottleneck. Many gradient compression (GC) algorithms have been proposed to address this communication bottleneck by reducing the amount of communicated data. Unfortunately, it has been observed that GC only achieves moderate performance improvement in DDT, or even harms the performance. In this paper, we argue that the current way of deploying GC in a layer-wise fashion reduces communication time but at the cost of non-negligible compression overheads. To address this problem, we propose Cupcake, a compression optimizer to fully unleash GC algorithms’ advantages in accelerating DDT. It applies GC algorithms in a fusion fashion and determines the provably optimal fusion strategy to maximize the training throughput of compression-enabled DDT jobs. Experimental evaluations show that GC algorithms with Cupcake can achieve up to 2.03× speedup in training throughput over training without GC, and up to 1.79× speedup over the state-of-the-art approaches of applying GC to DDT in a layer-wise fashion.