Optimized Non-contiguous MPI Datatype Communication for GPU Clusters: Design, Implementation and Evaluation with MVAPICH2
Optimized Non-contiguous MPI Datatype Communication for GPU Clusters: Design, Implementation and Evaluation with MVAPICH2
复制标题
针对 GPU 集群的优化非连续 MPI 数据类型通信：使用 MVAPICH2 进行设计、实现和评估
DOI：
10.1109/cluster.2011.42
复制
发表时间：
2011
期刊：
影响因子：
--
通讯作者：
D. Panda
中科院分区：
文献类型：
--
作者：
Hao Wang;S. Potluri;Miao Luo;A. Singh;Xiangyong Ouyang;S. Sur;D. Panda
数据并行体系结构，如通用图形单元(GPGPU)，在高端计算中的应用出现了巨大的增长。然而，进出GPGPU的数据移动仍然是整体性能和程序员工作效率的最大障碍。真正的科学应用利用多维数据。更高维度的数据在内存中可能不是连续的。为了提高程序员的工作效率并使通信库能够优化非连续数据通信，MPI接口提供了MPI数据类型。目前，最先进的MPI库不为驻留在GPU存储器中的数据提供本机数据类型支持。管理非连续的GPU数据是生产力和性能损失的一个来源，因为GPU应用程序开发人员必须手动将数据移出和移入GPU。在这篇文章中，我们介绍了我们的设计，以实现高性能的通信支持之间的图形处理器之间的非连续数据类型。我们描述了我们的创新方法，通过将数据类型打包和解包“卸载”到一个GPU设备上，并将两个GPU之间的所有数据传输阶段“流水线化”，从而提高性能。我们的设计集成到了用于InfiniBand、iWARP和ROCE集群的流行的MVAPICH2 MPI库中。我们在使用最新NVIDIA Fermi GPU适配器的GPU集群上对我们的设计进行了详细的评估。评估表明，对于4MB大小的向量数据类型，所提出的设计可以在微基准测试下获得高达88%的延迟改进。对于SHOC基准测试套件中的Stencil2D应用，我们的设计可以简化其主循环中的数据通信，减少36%的代码行。此外，对于单精度数据集，我们的方法可以将Stencil2D的性能提高42%，对于双精度数据集，我们的方法可以将性能提高39%。据我们所知，这是第一次为GPU集群设计、实现和评估非连续MPI数据通信。
Data parallel architectures, such as General Purpose Graphics Units (GPGPUs) have seen a tremendous rise in their application for High End Computing. However, data movement in and out of GPGPUs remains the biggest hurdle to overall performance and programmer productivity. Real scientific applications utilize multi-dimensional data. Data in higher dimensions may not be contiguous in memory. In order to improve programmer productivity and to enable communication libraries to optimize non-contiguous data communication, the MPI interface provides MPI data types. Currently, state of the art MPI libraries do not provide native data type support for data that resides in GPU memory. The management of non-contiguous GPU data is a source of productivity and performance loss, because GPU application developers have to manually move the data out of and in to GPUs. In this paper, we present our design for enabling high-performance communication support between GPUs for non-contiguous data types. We describe our innovative approach to improve performance by "offloading" data type packing and unpacking on to a GPU device, and "pipelining" all data transfer stages between two GPUs. Our design is integrated into the popular MVAPICH2 MPI library for InfiniBand, iWARP and RoCE clusters. We perform a detailed evaluation of our design on a GPU cluster with the latest NVIDIA Fermi GPU adapters. The evaluation reveals that the proposed designs can achieve up to 88% latency improvement for vector data type at 4 MB size with micro benchmarks. For Stencil2D application from the SHOC benchmark suite, our design can simplify the data communication in its main loop, reducing the lines of code by 36%. Further, our method can improve the performance of Stencil2D by up to 42% for single precision data set, and 39% for double precision data set. To the best of our knowledge, this is the first such design, implementation and evaluation of non-contiguous MPI data communication for GPU clusters.