Gradient Compression Supercharged High-Performance Data Parallel DNN Training
Gradient Compression Supercharged High-Performance Data Parallel DNN Training
复制标题
DOI：
10.1145/3477132.3483553
复制
发表时间：
2021-10
期刊：
影响因子：
--
通讯作者：
Youhui Bai;Cheng Li;Quan Zhou;Jun Yi;Ping Gong;Feng Yan;Ruichuan Chen;Yinlong Xu
中科院分区：
文献类型：
--
作者：
Youhui Bai;Cheng Li;Quan Zhou;Jun Yi;Ping Gong;Feng Yan;Ruichuan Chen;Yinlong Xu
梯度压缩是一种很有前途的方法，可以通过显着减少同步梯度的数据量来缓解数据并行深度神经网络（DNN）训练中的通信瓶颈。虽然梯度压缩正在被业界（例如 Facebook 和 AWS）积极采用，但我们的研究表明，存在两个关键但经常被忽视的挑战：1）梯度同步期间压缩和通信之间的低效协调会产生大量开销，2）开发、优化梯度压缩算法并将其集成到 DNN 系统中给 DNN 从业者带来沉重负担，而临时压缩实现通常会产生令人惊讶的糟糕系统性能。在本文中，我们首先提出了一种压缩感知梯度同步架构CaSync，它依赖于基本计算和通信原语的灵活组合。它具有通用性，兼容任何梯度压缩算法和梯度同步策略，并支持高性能计算通信流水线。我们进一步引入了梯度压缩工具包 CompLL，以实现 GPU 上压缩算法的高效开发和自动集成到 DNN 系统中，而编程负担很小。最后，我们使用 CaSync 和 CompLL 构建了一个压缩感知 DNN 训练框架 HiPress。 HiPress 是开源的，运行在 MXNet、TensorFlow 和 PyTorch 等主流 DNN 系统上。通过具有 128 个 NVIDIA V100 GPU 和 100Gbps 网络的 16 节点集群进行的评估表明，在六种流行的 DNN 模型中，HiPress 将当前支持压缩的系统（例如 BytePS-onebit 和 Ring-DGC）的训练速度提高了 17.2%-69.5%。
Gradient compression is a promising approach to alleviating the communication bottleneck in data parallel deep neural network (DNN) training by significantly reducing the data volume of gradients for synchronization. While gradient compression is being actively adopted by the industry (e.g., Facebook and AWS), our study reveals that there are two critical but often overlooked challenges: 1) inefficient coordination between compression and communication during gradient synchronization incurs substantial overheads, and 2) developing, optimizing, and integrating gradient compression algorithms into DNN systems imposes heavy burdens on DNN practitioners, and ad-hoc compression implementations often yield surprisingly poor system performance. In this paper, we first propose a compression-aware gradient synchronization architecture, CaSync, which relies on a flexible composition of basic computing and communication primitives. It is general and compatible with any gradient compression algorithms and gradient synchronization strategies, and enables high-performance computation-communication pipelining. We further introduce a gradient compression toolkit, CompLL, to enable efficient development and automated integration of on-GPU compression algorithms into DNN systems with little programming burden. Lastly, we build a compression-aware DNN training framework HiPress with CaSync and CompLL. HiPress is open-sourced and runs on mainstream DNN systems such as MXNet, TensorFlow, and PyTorch. Evaluation via a 16-node cluster with 128 NVIDIA V100 GPUs and 100Gbps network shows that HiPress improves the training speed over current compression-enabled systems (e.g., BytePS-onebit and Ring-DGC) by 17.2%-69.5% across six popular DNN models.