Collaborative Research: Frameworks: Performance Engineering Scientific Applications with MVAPICH and TAU using Emerging Communication Primitives

地震灾害对社区构成潜在的威胁，并造成重大的经济损失。大型超级计算机上的地震数值模拟正在成为指导基础设施和政策决策的关键，这是地震建模的结果。这些地震和其他代码，包括涉及快速傅里叶变换（FFT）的模拟，将处理分布在超级计算机中的大量计算节点上。优化节点之间的通信是实现良好性能的关键，但考虑到执行的规模，这是一项艰巨的任务。实现消息传递接口（MPI）的MVAPICH通信库和用于观察通信的分析工具TAU性能系统将紧密耦合在一起，以评估在执行过程中调整这些代码对性能的影响。这些库将共享关键性能参数，并优化这些应用程序中的通信，以缩短解决方案的时间。这些代码的性能工程版本将有助于推动下一代地震预报，并有助于提高我们对地震事件的理解，以减少对人口中心和环境的风险。该研究将通过研究高性能计算（HPC）、深度/机器学习和数据分析课程的教学法，推动本科和研究生课程的进步。研究结果也将分发给研究人员的合作组织，以影响他们的高性能计算软件应用。新兴的高性能计算系统——由多核处理器和加速器架构驱动——需要在现有基础设施上进行创新，以为科学领域提供最佳性能。MPI 4.0标准也为协同设计应用程序提供了新的机会。这些包括分区的点对点和集体行动，以及邻里集体行动。随着这些进步，迫切需要更新常用的工具和库，这些工具和库构成了NSF高性能计算网络基础设施的基础。该研究承担了这一挑战，并通过利用MPI_T API的协同设计方法，在MVAPICH2和TAU库中进行科学应用，追求新的性能工程途径。该项目重点关注两种流行的高性能计算应用，它们跨越多个领域，代表各种通信模式——非弹性波传播（AWP-ODC）和用于百亿亿级的高效fft （heFFTe）。AWP-ODC是一个高度可扩展的并行有限差分应用程序，具有点对点操作，可实现三维地震计算。HeFFTe是一种大规模并行应用程序，以集合运算为主导，为广泛使用的快速傅里叶变换（FFT）运算提供了可扩展和高效的实现。该研究旨在通过共同设计MVAPICH2和TAU库来研究和开发以下创新，以扩展驾驶科学领域-包括AWP-ODC和heFFTe：1) MPI异步通信的负载感知设计，2)MPI+X应用程序的跨运行时协调，3)分区点对点原语，4)应用感知的邻域集体通信，5)支持自适应持久集体通信，6)gpu上的协调通信内核。集成开发和评估的进行，以确保建议的设计与驱动应用程序的适当集成，并与内部和外部合作者密切合作，以促进广泛部署和采用发布的软件。所提出的工作的变革性影响是通过智能性能工程在下一代HPC架构中提取高性能计算应用程序的性能和可扩展性。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Earthquake hazards pose potentially life-threatening risks to communities and cause significant economic damage. Numerical simulations of earthquakes on large-scale supercomputers are emerging as key to guiding the infrastructure and policy decisions as a result of earthquake modeling. These seismic and other codes including simulations involving Fast Fourier Transform (FFT) distribute the processing across a large number of compute nodes in a supercomputer. Optimizing the communication between nodes is key to achieving good performance but it is a daunting task given the scale of execution. The MVAPICH communication library that implements the Message Passing Interface (MPI) and the TAU Performance System, a profiling tool to observe the communication, will be tightly coupled to assess the performance impact of tuning these codes during execution. These libraries will share key performance parameters and optimize the communication in these applications to improve the time to solution. Performance-engineered versions of these codes will help drive the next generation of earthquake forecasting and help improve our understanding of seismic events to reduce risks to population centers and the environment. The research will enable undergraduate and graduate curriculum advancements via research in pedagogy for High Performance Computing (HPC), Deep/Machine Learning, and Data Analytics courses. The results will also be disseminated to the collaborating organizations of the investigators to impact their HPC software applications.Emerging HPC systems---driven by many-core processors and accelerator architectures--- require innovations in existing infrastructure to deliver the best performance for science domains. The MPI 4.0 standard has also brought forward new opportunities for co-designing applications. These include partitioned point-to-point and collective operations, and neighborhood collectives. With these advances, there is a critical need to update the commonly used tools and libraries that form the basis for the NSF’s HPC cyberinfrastructure. The research undertakes this challenge and pursues new performance engineering avenues---by exploiting a co-design approach using the MPI_T API---in the MVAPICH2 and TAU libraries with scientific applications. The project focuses on two popular HPC applications spanning multiple domains and representing various communication patterns - Anelastic Wave Propagation (AWP-ODC) and Highly efficient FFTs for Exascale (heFFTe). AWP-ODC is a highly scalable parallel finite-difference application with point-to-point operations that enables 3D earthquake calculations. HeFFTe, dominated by collective operations, is a massively parallel application that provides a scalable and efficient implementation of the widely used Fast Fourier Transform (FFT) operations. The research aims to investigate and develop the following innovations by co-designing MVAPICH2 and TAU libraries to scale driving science domains---including AWP-ODC and heFFTe: 1) Load-aware designs for MPI asynchronous communication, 2) Cross runtime coordination for MPI+X applications, 3) Partitioned point-to-point primitives, 4) Application-aware neighborhood collective communication, 5) Support for adaptive persistent collective communication, and 6) Coordinating communication kernels on GPUs. Integrated development and evaluation are carried out to ensure proper integration of proposed designs with the driving applications, and closely work with internal and external collaborators to facilitate wide deployment and adoption of the released software. The transformative impact of the proposed effort is to extract the performance and scalability of HPC applications in next-generation HPC architectures through intelligent performance engineering.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.