Designing Hierarchical Multi-HCA Aware Allgather in MPI
Designing Hierarchical Multi-HCA Aware Allgather in MPI
复制标题
DOI：
10.1145/3547276.3548524
复制
发表时间：
2022-08
期刊：
影响因子：
--
通讯作者：
Tu Tran;Benjamin Michalowicz;B. Ramesh;H. Subramoni;A. Shafi;D. Panda
中科院分区：
文献类型：
--
作者：
Tu Tran;Benjamin Michalowicz;B. Ramesh;H. Subramoni;A. Shafi;D. Panda
为了加速节点之间的通信，超级计算机现在为每个节点配备多个网络适配器，形成 "多轨 "网络。Top500 排名第二和第三的系统每个节点使用两个适配器；最近，阿贡国家实验室（ANL）的 ThetaGPU 系统每个节点使用八个适配器。在网络资源如此充足的情况下，要充分利用所有网络资源并非易事。消息传递接口（MPI）是高性能计算集群的主流模式。并不是所有的 MPI 集合都能利用所有资源，这一点随着特定集群带宽和适配器数量的增加而变得更加明显。在这项工作中，我们承担了这项任务，并提出了分层、多HCA 感知的 Allgather 设计；Allgather 是一种通信密集型集群，广泛应用于矩阵乘法和其他集群。所提出的设计充分利用了节点内所有可用的网络适配器，并提供了节点间和节点内通信的高度重叠。在微基准层面，我们的新方案实现了单节点和多节点通信的性能提升。在 1024 个进程的情况下，我们发现节点间的性能比 HPC-X 和 MVAPICH2-X 分别提高了 62% 和 61%。与 HPC-X 和 MVAPICH2-X 相比，节点间通信设计还将环形 Allreduce 的性能分别提高了 56% 和 44%。在应用层面，与 HPC-X 和 MVAPICH2-X 相比，增强型 Allgather 在矩阵向量乘法内核中的性能分别提高了 1.98 倍和 1.42 倍；与 MVAPICH2-X 相比，Allreduce 在深度学习训练中的性能提高了 7.83%。
To accelerate the communication between nodes, supercomputers are now equipped with multiple network adapters per node, resulting in a ”multi-rail” network. The second and third-placed systems of the Top500 use two adapters per node; recently, the ThetaGPU system at Argonne National Laboratory (ANL) uses eight adapters per node. With such an availability of networking resources, it is a non-trivial task to utilize all of them. The Message Passing Interface (MPI) is a dominant model for high-performance computing clusters. Not all MPI collectives utilize all resources, and this becomes more apparent with advances in bandwidth and adapter count in a given cluster. In this work, we take up this task and propose hierarchical, multi-HCA aware Allgather designs; Allgather is a communication-intensive collective widely used in applications like matrix multiplication and other collectives. The proposed designs fully utilize all the available network adapters within a node and provides high overlap between inter-node and intra-node communication. At the micro-benchmark level, our new schemes achieve performance improvement for both single node and multiple node communication. We see inter-node improvements up to 62% and 61% better than HPC-X and MVAPICH2-X for 1024 processes. The design for inter-node communication also boosts the performance of Ring Allreduce by 56% and 44% compared to HPC-X and MVAPICH2-X. At the application level, the enhanced Allgather shows 1.98x and 1.42x improvement in a matrix-vector multiplication kernel when compared to HPC-X and MVAPICH2-X, and Allreduce performs up to 7.83% better in deep learning training against MVAPICH2-X.