Designing Non-blocking Allreduce with Collective Offload on InfiniBand Clusters: A Case Study with Conjugate Gradient Solvers
Designing Non-blocking Allreduce with Collective Offload on InfiniBand Clusters: A Case Study with Conjugate Gradient Solvers
复制标题
在 InfiniBand 集群上设计具有集体卸载的非阻塞 Allreduce：共轭梯度求解器的案例研究
DOI：
10.1109/ipdps.2012.106
复制
发表时间：
2011
期刊：
影响因子：
--
通讯作者：
D. Panda
中科院分区：
文献类型：
--
作者：
K. Kandalla;U. Yang;J. Keasler;T. Kolev;A. Moody;H. Subramoni;K. Tomko;Jérôme Vienne;B. Supinski;D. Panda
许多领域的科学家越来越多地依赖计算机模拟进行研究。这种模拟通常花费大部分运行时间来求解大型线性方程组，这需要大量的计算能力和内存。因此，以高效和可扩展的方式设计求解器至关重要。Hypre是一个高性能、可扩展的软件库，提供了几个优化的线性求解器例程和预处理器。本文研究了Hypre的预条件共轭梯度（PCG）求解算法的特点。众所周知，PCG例程将其大部分通信时间花费在MPI All归约操作中，以在内积操作期间计算全局求和。MPI All reduce是一个阻塞操作，其延迟通常是PCG求解器例程的整体效率的限制因素，并且相应地限制依赖于此求解器的模拟的性能。因此，隐藏MPI All reduce操作的延迟对于扩展PCG求解器例程和提高许多模拟的性能至关重要。即将发布的MPI修订版MPI-3将支持非阻塞集体通信，以实现延迟隐藏。来自Mellanox的最新InfiniBand适配器ConnectX-2可以将通信操作的通用列表卸载到网络接口。这样的接口可以用来设计非阻塞的集体操作。在本文中，我们设计了功能齐全，可扩展的算法的MPI_Iallreduce操作，基于网络卸载技术。据我们所知，这些基于网络卸载的算法是首次为MPI_Iallreduce操作提出的。我们的设计规模超过512个进程，我们实现了近乎完美的通信/计算重叠。我们还重新设计了PCG求解器例程，以利用我们提出的MPI_Iallreduce操作来隐藏全局约简操作的延迟。我们观察到高达21%的改进，在运行时的PCG例程，相比默认的PCG实现在Hypre。我们还注意到，约16%的整体收益是由于重叠的所有减少操作。
Scientists across a wide range of domains increasingly rely on computer simulation for their investigations. Such simulations often spend a majority of their run-times solving large systems of linear equations that require vast amounts of computational power and memory. It is hence critical to design solvers in a highly efficient and scalable manner. Hypre is a high performance, scalable software library that offers several optimized linear solver routines and pre-conditioners. In this paper, we study the characteristics of Hypre's Preconditioned Conjugate Gradient (PCG) solver algorithm. The PCG routine is known to spend a majority of its communication time in the MPI All reduce operation to compute a global summation during the inner product operation. The MPI All reduce is a blocking operation, whose latency is often a limiting factor to the overall efficiency of the PCG solver routine, and correspondingly the performance of simulations that rely on this solver. Hence, hiding the latency of the MPI All reduce operation is critical towards scaling the PCG solver routine and improving the performance of many simulations. The upcoming revision of MPI, MPI-3, will provide support for non-blocking collective communication to enable latency-hiding. The latest InfiniBand adapter from Mellanox, ConnectX-2, enables offloading of generalized lists of communication operations to the network interface. Such an interface can be leveraged to design non-blocking collective operations. In this paper, we design fully functional, scalable algorithms for the MPI_Iallreduce operation, based on the network offload technology. To the best of our knowledge, these network offload-based algorithms are the first to be presented for the MPI_Iallreduce operation. Our designs scale beyond 512 processes and we achieve near perfect communication/computation overlap. We also re-design the PCG solver routine to leverage our proposed MPI_Iallreduce operation to hide the latency of the global reduction operations. We observe up to 21% improvements in the run-times of the PCG routine, when compared to the default PCG implementation in Hypre. We also note that about 16% of the overall benefits are due to overlapping the All reduce operations.