Collaborative Research: Frameworks: Performance Engineering Scientific Applications with MVAPICH and TAU using Emerging Communication Primitives

地震灾害对社区构成潜在的生命威胁，并造成重大经济损失。大型超级计算机上的地震数值模拟正在成为指导地震建模基础设施和政策决策的关键。这些地震和其他代码，包括涉及快速傅立叶变换（FFT）的模拟，将处理分布在超级计算机中的大量计算节点上。优化节点之间的通信是实现良好性能的关键，但考虑到执行的规模，这是一项艰巨的任务。实现消息传递接口（MPI）的MVAPICH通信库和TAU性能系统（用于观察通信的分析工具）将紧密耦合，以评估在执行期间调整这些代码的性能影响。这些库将共享关键性能参数，并优化这些应用程序中的通信，以缩短解决方案的时间。这些代码的性能工程版本将有助于推动下一代地震预报，并有助于提高我们对地震事件的理解，以减少对人口中心和环境的风险。 该研究将通过高性能计算（HPC），深度/机器学习和数据分析课程的教学研究，实现本科生和研究生课程的进步。研究结果还将分发给研究人员的合作组织，以影响他们的HPC软件应用程序。新兴的HPC系统--由众核处理器和加速器架构驱动--需要在现有基础设施中进行创新，以便为科学领域提供最佳性能。MPI 4.0标准也为协同设计应用程序带来了新的机遇。这些包括分区的点对点和集体操作，以及邻域集体。随着这些进步，迫切需要更新构成NSF HPC网络基础设施基础的常用工具和库。研究承担这一挑战，并追求新的性能工程途径-通过利用MPI_T API的协同设计方法-在MVAPICH 2和TAU库与科学应用。该项目的重点是两个流行的HPC应用程序跨越多个领域，代表各种通信模式-滞弹性波传播（AWP-ODC）和Exascale高效FFT（heFFTe）。AWP-ODC是一个高度可扩展的并行有限差分应用程序，具有点对点操作，可以进行3D地震计算。HeFFTe是一个大规模并行的应用程序，它提供了一个可扩展的和有效的实现广泛使用的快速傅立叶变换（FFT）操作。该研究旨在通过共同设计MVAPICH 2和TAU库来扩展驾驶科学领域-包括AWP-ODC和heFFTe，以研究和开发以下创新：1）MPI异步通信的负载感知设计，2）MPI+X应用的跨运行时协调，3）分区点对点原语，4）应用感知邻域集体通信，5）支持自适应持久集体通信，以及6）协调GPU上的通信内核。进行综合开发和评估，以确保建议的设计与驱动应用程序适当整合，并与内部和外部合作者密切合作，以促进广泛部署和采用已发布的软件。该奖项旨在通过智能性能工程来提升下一代HPC架构中HPC应用程序的性能和可扩展性。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响评审标准进行评估，被认为值得支持。

Earthquake hazards pose potentially life-threatening risks to communities and cause significant economic damage. Numerical simulations of earthquakes on large-scale supercomputers are emerging as key to guiding the infrastructure and policy decisions as a result of earthquake modeling. These seismic and other codes including simulations involving Fast Fourier Transform (FFT) distribute the processing across a large number of compute nodes in a supercomputer. Optimizing the communication between nodes is key to achieving good performance but it is a daunting task given the scale of execution. The MVAPICH communication library that implements the Message Passing Interface (MPI) and the TAU Performance System, a profiling tool to observe the communication, will be tightly coupled to assess the performance impact of tuning these codes during execution. These libraries will share key performance parameters and optimize the communication in these applications to improve the time to solution. Performance-engineered versions of these codes will help drive the next generation of earthquake forecasting and help improve our understanding of seismic events to reduce risks to population centers and the environment. The research will enable undergraduate and graduate curriculum advancements via research in pedagogy for High Performance Computing (HPC), Deep/Machine Learning, and Data Analytics courses. The results will also be disseminated to the collaborating organizations of the investigators to impact their HPC software applications.Emerging HPC systems---driven by many-core processors and accelerator architectures--- require innovations in existing infrastructure to deliver the best performance for science domains. The MPI 4.0 standard has also brought forward new opportunities for co-designing applications. These include partitioned point-to-point and collective operations, and neighborhood collectives. With these advances, there is a critical need to update the commonly used tools and libraries that form the basis for the NSF’s HPC cyberinfrastructure. The research undertakes this challenge and pursues new performance engineering avenues---by exploiting a co-design approach using the MPI_T API---in the MVAPICH2 and TAU libraries with scientific applications. The project focuses on two popular HPC applications spanning multiple domains and representing various communication patterns - Anelastic Wave Propagation (AWP-ODC) and Highly efficient FFTs for Exascale (heFFTe). AWP-ODC is a highly scalable parallel finite-difference application with point-to-point operations that enables 3D earthquake calculations. HeFFTe, dominated by collective operations, is a massively parallel application that provides a scalable and efficient implementation of the widely used Fast Fourier Transform (FFT) operations. The research aims to investigate and develop the following innovations by co-designing MVAPICH2 and TAU libraries to scale driving science domains---including AWP-ODC and heFFTe: 1) Load-aware designs for MPI asynchronous communication, 2) Cross runtime coordination for MPI+X applications, 3) Partitioned point-to-point primitives, 4) Application-aware neighborhood collective communication, 5) Support for adaptive persistent collective communication, and 6) Coordinating communication kernels on GPUs. Integrated development and evaluation are carried out to ensure proper integration of proposed designs with the driving applications, and closely work with internal and external collaborators to facilitate wide deployment and adoption of the released software. The transformative impact of the proposed effort is to extract the performance and scalability of HPC applications in next-generation HPC architectures through intelligent performance engineering.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.