Accelerating MPI All-to-All Communication with Online Compression on Modern GPU Clusters
Accelerating MPI All-to-All Communication with Online Compression on Modern GPU Clusters
复制标题
在现代 GPU 集群上通过在线压缩加速 MPI 全面通信
DOI：
10.1007/978-3-031-07312-0_1
复制
发表时间：
2022
期刊：
影响因子：
--
通讯作者：
Panda, DK.
中科院分区：
文献类型：
--
作者：
Zhou, Q.;Kousha, P.;Anthony, Q.;Khorassani, K.;Shafi, A.;Subramoni, H.;Panda, DK.
随着越来越多的高性能计算（HPC）和深度学习（DL）应用程序适应使用gpu进行扩展，gpu驻留数据的通信对端到端应用程序性能变得至关重要。在这些应用程序中可用的MPI操作中，All-to-All是通信最密集的操作之一，它成为有效地将应用程序扩展到更大的GPU系统的瓶颈。在过去的十年中，大多数研究都集中在优化大型gpu驻留数据传输上。然而，对于最先进的gpu感知MPI库，由于商品网络的吞吐量限制，用于大型gpu驻留数据的MPI_Alltoall通信仍然受到性能差的影响。然而，基于gpu的高吞吐量压缩算法的发展可以减少数据传输量。基于点对点的在线压缩的最新研究表明，这些压缩算法在现代GPU集群上具有很大的潜力。在本文中，我们重新设计了一个MPI库，以实现高效的集体级在线压缩，并优化了所有对所有通信的主机分级方案。我们证明了所提出的设计在微基准测试和应用程序级别上都有好处。在微基准测试水平上，所提出的设计可以将All-to-All通信延迟降低高达87%。对于传统的高性能计算应用PSDNS，在保证数据验证和不影响应用收敛时间的情况下，我们提出的设计可以将All-to-All通信延迟和总运行时间分别降低29.2%和21.8%。对于微软的深度学习优化库DeepSpeed，与最先进的MPI库相比，该设计在确保数据验证的同时，将MPI_Alltoall运行时间减少了26.4%。据我们所知，这是第一个利用基于gpu的在线压缩技术来显著加速HPC和DL应用程序的MPI_Alltoall通信的工作。
As more High-Performance Computing (HPC) and Deep Learning (DL) applications are adapting to scale using GPUs, the communication of GPU-resident data is becoming vital to end-to-end application performance. Among the available MPI operations in such applications, All-to-All is one of the most communication-intensive operations that becomes the bottleneck of efficiently scaling applications to larger GPU systems. Over the last decade, most research has focused on the optimization of large GPU-resident data transfers. However, for state-of-the-art GPU-Aware MPI libraries, MPI_Alltoall communication for large GPU-resident data still suffers from poor performance due to the throughput limitation of commodity networks. However, the development of GPU-based compression algorithms with high throughput can reduce the volume of data transferred. The recent research of point-to-point-based online compression with these compression algorithms has shown potential on modern GPU clusters.In this paper, we redesign an MPI library to enable efficient collective-level online compression with an optimized host-staging scheme for All-to-All communication. We demonstrate that the proposed design achieves benefits at both microbenchmark and application levels. At the microbenchmark level, the proposed design can reduce the All-to-All communication latency by up to 87%. For PSDNS, a traditional HPC application, our proposed design can reduce the All-to-All communication latency and total runtime by up to 29.2% and 21.8%, respectively, while ensuring data validation and not affecting the application convergence time. For Microsoft’s DeepSpeed, a DL optimization library, the proposed design reduces the MPI_Alltoall runtime by up to 26.4% compared to a state-of-the-art MPI library with point-to-point compression while ensuring data validation. To the best of our knowledge, this is the first work that leverages online GPU-based compression techniques to significantly accelerate MPI_Alltoall communication for HPC and DL applications.