Understanding GPU Triggering APIs for MPI+X Communication

GPU-enhanced architectures are now dominant in HPC systems, but message-passing communication involving GPUs with MPI has proven to be both complex and expensive, motivating new approaches that lower such costs. We compare and contrast stream/graph- and kernel-triggered MPI communication abstractions, whose principal purpose is to enhance the performance of communication when GPU kernels create or consume data for transfer through MPI operations. Researchers and practitioners have proposed multiple potential APIs for stream and/or kernel triggering that span various GPU architectures and approaches, including MPI-4 partitioned point-to-point communication, stream communicators, and explicit MPI stream/queue objects. Designs breaking backward compatibility with MPI are duly noted. Some of these strengthen or weaken the semantics of MPI operations. A key contribution of this paper is to promote community convergence toward a stream- and/or kernel-triggering abstraction by highlighting the common and differing goals and contributions of existing abstractions. We describe the design space in which these abstractions reside, their implicit or explicit use of stream and other non-MPI abstractions, their relationship to partitioned and persistent operations, and discuss their potential for added performance, how usable these abstractions are, and where functional and/or semantic gaps exist. Finally, we provide a taxonomy for stream- and kernel-triggered abstractions, including disambiguation of similar semantic terms, and consider directions for future standardization in MPI-5.

GPU增强型架构现在在HPC系统中占主导地位，但涉及GPU与MPI的消息传递通信已被证明既复杂又昂贵，从而激发了降低此类成本的新方法。我们比较和对比流/图形和内核触发的MPI通信抽象，其主要目的是提高通信的性能时，GPU内核创建或消费数据传输通过MPI操作。研究人员和实践者已经提出了多种潜在的API用于流和/或内核触发，这些API跨越各种GPU架构和方法，包括MPI-4分区点对点通信、流通信器和显式MPI流/队列对象。破坏MPI向后兼容性的设计也被适当地注意到。其中一些增强或削弱了MPI操作的语义。本文的一个关键贡献是通过突出现有抽象的共同和不同的目标和贡献，促进社区向流和/或内核触发抽象的融合。我们描述的设计空间，这些抽象驻留，其隐式或显式使用流和其他非MPI抽象，其关系分区和持久性操作，并讨论其潜在的增加性能，如何使用这些抽象，以及功能和/或语义的差距存在。最后，我们提供了一个分类的流和内核触发的抽象，包括消歧相似的语义术语，并考虑在MPI-5的未来标准化的方向。