Understanding GPU Triggering APIs for MPI+X Communication

GPU-enhanced architectures are now dominant in HPC systems, but message-passing communication involving GPUs with MPI has proven to be both complex and expensive, motivating new approaches that lower such costs. We compare and contrast stream/graph- and kernel-triggered MPI communication abstractions, whose principal purpose is to enhance the performance of communication when GPU kernels create or consume data for transfer through MPI operations. Researchers and practitioners have proposed multiple potential APIs for stream and/or kernel triggering that span various GPU architectures and approaches, including MPI-4 partitioned point-to-point communication, stream communicators, and explicit MPI stream/queue objects. Designs breaking backward compatibility with MPI are duly noted. Some of these strengthen or weaken the semantics of MPI operations. A key contribution of this paper is to promote community convergence toward a stream- and/or kernel-triggering abstraction by highlighting the common and differing goals and contributions of existing abstractions. We describe the design space in which these abstractions reside, their implicit or explicit use of stream and other non-MPI abstractions, their relationship to partitioned and persistent operations, and discuss their potential for added performance, how usable these abstractions are, and where functional and/or semantic gaps exist. Finally, we provide a taxonomy for stream- and kernel-triggered abstractions, including disambiguation of similar semantic terms, and consider directions for future standardization in MPI-5.

GPU增强架构如今在高性能计算（HPC）系统中占据主导地位，但事实证明，使用消息传递接口（MPI）在GPU之间进行消息传递通信既复杂又昂贵，这促使人们寻求降低此类成本的新方法。我们对基于流/图以及由内核触发的MPI通信抽象进行了比较和对比，其主要目的是在GPU内核通过MPI操作创建或使用数据进行传输时提高通信性能。研究人员和实践者针对流和/或内核触发提出了多种潜在的应用程序编程接口（API），它们涵盖了各种GPU架构和方法，包括MPI - 4分区的点对点通信、流通信器以及显式的MPI流/队列对象。我们适当地指出了那些破坏与MPI向后兼容性的设计。其中一些设计强化或弱化了MPI操作的语义。本文的一个关键贡献是通过强调现有抽象的共同及不同目标和贡献，推动社区朝着基于流和/或内核触发的抽象趋同。我们描述了这些抽象所处的设计空间、它们对流和其他非MPI抽象的隐式或显式使用、它们与分区和持久化操作的关系，并讨论了它们在提高性能方面的潜力、这些抽象的可用性以及存在功能和/或语义差距的地方。最后，我们为基于流和内核触发的抽象提供了一种分类法，包括对相似语义术语的辨析，并考虑了MPI - 5未来标准化的方向。