Design of a Portable Implementation of Partitioned Point-to-Point Communication Primitives
Design of a Portable Implementation of Partitioned Point-to-Point Communication Primitives
复制标题
分区点对点通信原语的便携式实现的设计
DOI：
10.1145/3458744.3474046
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Ghafoor, Sheikh
中科院分区：
文献类型：
--
作者：
Worley, Andrew;Prema Soundararajan, Prema;Schafer, Derek;Bangalore, Purushotham;Grant, Ryan;Dosanjh, Matthew;Skjellum, Anthony;Ghafoor, Sheikh
几十年来，消息传递接口 (MPI) 一直是科学计算的主要消息传递解决方案。 MPI 点对点通信是进程间通信的高效机制。但是，当进程使用多个线程时，MPI 库中的并发保护会降低 MPI 性能。当需要线程安全时，MPI 当前的线程级接口会在整个库中施加这些开销。虽然已经做了很多工作来减少 MPI 中的多线程开销，但仍需要一种解决方案来减少线程环境中交换的消息数量。分区通信包含在 MPI 4.0 标准中，作为解决当今 MPI 中多线程通信挑战的替代方案。分区通信通过在线程之间创建缓冲区共享机制来减少总体消息量，以便线程可以指示通信缓冲区的某些部分何时可用于发送。 MPI 中控制平面和数据平面的分离是通过允许持久初始化和单次消息缓冲区与数据已准备好发送的指示进行匹配来实现的。这使得能够使用底层硬件原语，例如触发操作，其中可以在数据缓冲区准备就绪之前设置命令（目的地、大小等），并在稍后由简单的门铃/计数器触发准备就绪。这种方法对于传统网络命令可能面临性能挑战的环境（例如加速器（GPU、FPGA））中的 MPI 操作的未来开发非常有用。在本文中，我们详细介绍了分层库（构建在 MPI-3.1 之上）和支持新的 MPI-4.0 分区通信功能集的集成 Open MPI 解决方案的设计和实现。该库将使应用程序能够使用当前发布的 MPI 实现和较旧的遗留库来提供分区通信支持，同时还可以在新应用程序和用例中进一步探索这种新的通信模型。我们将比较库和本机 Open MPI 支持的设计，提供两种方法之间的性能结果和比较，以及从库和本机形式的分区通信实现中吸取的经验教训。我们发现本机实现和库具有相似的性能，在微基准测试中百分比差异低于 0.94%，对于支持分区通信的代理应用程序，性能差异在 5% 以内。
The Message Passing Interface (MPI) has been the dominant message passing solution for scientific computing for decades. MPI point-to-point communications are highly efficient mechanisms for process-to-process communication. However, MPI performance is slowed by concurrency protections in the MPI library when processes utilize multiple threads. MPI’s current thread-level interface imposes these overheads throughout the library when thread safety is needed. While much work has been done to reduce multithreading overheads in MPI, a solution is needed that reduces the number of messages exchanged in a threaded environment.Partitioned communication is included in the MPI 4.0 standard as an alternative that addresses the challenges of multithreaded communication in MPI today. Partitioned communication reduces overall message volume by creating a buffer-sharing mechanism between threads such that they can indicate when portions of a communication buffer are available to be sent. Separation of the control and data planes in MPI is enabled by allowing persistent initialization and single occurrence message buffer matching from the indication that the data is ready to be sent. This enables the usage of underlying hardware primitives like triggered operations, where commands (destination, size, etc.) can be set up prior to data buffer readiness with readiness triggered by a simple doorbell/counter later. This approach is useful for future development of MPI operations in environments where traditional networking commands can have performance challenges, like accelerators (GPUs, FPGAs).In this paper, we detail the design and implementation of a layered library (built on top of MPI-3.1) and an integrated Open MPI solution that supports the new, MPI-4.0 partitioned communication feature set. The library will enable applications to use currently released MPI implementations and older legacy libraries to provide partitioned communication support while also enabling further exploration of this new communication model in new applications and use cases. We will compare the designs of the library and native Open MPI support, provide performance results and comparisons between the two approaches, and lessons learned from the implementation of partitioned communication in both library and native forms.We find that the native implementation and library have similar performance with a percentage difference under 0.94% in microbenchmarks and performance within 5% for a partitioned communication enabled proxy application.