Collaborative Research: Frameworks: Designing Next-Generation MPI Libraries for Emerging Dense GPU Systems

现代图形处理单元（GPU）和高性能互连提供的极高计算和通信能力已导致创建每个节点具有多个GPU和高性能互连的高性能计算（HPC）平台。不幸的是，流行的消息传递接口（MPI）编程模型的最先进的生产质量实现没有适当的支持，以提供最佳的性能和可扩展性的应用程序在这种密集的GPU系统。高端计算（HEC）技术和相关中间件问题的这些发展导致了以下广泛的挑战：如何增强现有的生产质量MPI中间件，以利用新兴的网络技术，为新兴的密集GPU系统上的HPC和深度学习（DL）应用提供最佳的纵向扩展和横向扩展？一个协同和全面的研究计划，涉及计算机科学家从俄亥俄州州立大学（OSU）和俄亥俄州超级计算机中心（OSC）和计算科学家从得克萨斯州高级计算中心（TACC），圣地亚哥超级计算机中心（SDSC）和加州圣地亚哥大学（UCSD），提出了解决上述广泛的挑战与创新的解决方案。拟议的框架将提供给合作者和更广泛的科学界，以了解拟议的创新对下一代HPC和DL框架以及各个科学领域应用的影响。 多个研究生和本科生将在这个项目下培养未来的科学家和工程师在HPC。拟议的工作将通过OSU，SDSC和TACC新数据科学计划中的关键课程的教学法研究实现课程进步。在TACC、SDSC和OSC建立的全国范围的培训和推广计划将用于向XSEDE用户传播这项研究的结果。在PEARC、SC和其他会议上将组织讲座和研讨会，与社区分享研究成果和经验。该项目与国家战略计算计划（NSCI）保持一致，以推动美国在HPC领域的领导地位，以及美国政府最近提出的保持人工智能（AI）领导地位的倡议。拟议的创新包括：1）设计高性能和可扩展的点对点和集体通信操作，充分利用多个网络适配器和先进的网络计算功能，用于节点内和节点间的GPU和CPU缓冲区; 2）设计新颖的数据类型处理和统一的内存管理，以提高应用性能; 3）设计CUDA感知的I/O子系统，加速HPC和DL应用的MPI I/O和检查点重启; 4）设计对容器化环境的支持，以更好地在现代云环境中轻松部署建议的解决方案;以及5）进行综合开发和评估，以确保建议的设计与驱动应用程序的适当集成。拟议的设计将被集成到广泛使用的MVAPICH 2库中并提供。项目小组成员将与内部和外部合作者密切合作，以促进广泛部署和采用已发布的软件。提出的解决方案将有针对性地在新兴的密集GPU平台上实现驱动科学领域（分子动力学、晶格QCD、地震学、图像分类和融合研究）的纵向扩展和横向扩展。该项目的发展成果将带来变革性的影响，即实现HPC和DL框架及应用程序的可扩展性、性能和可移植性，从而充分利用新兴的高密度GPU平台，从而推动科学和工程领域的重大进步。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响评审标准进行评估，被认为值得支持。

The extremely high compute and communication capabilities offered by modern Graphics Processing Units (GPUs) and high-performance interconnects have led to the creation of High-Performance Computing (HPC) platforms with multiple GPUs and high-performance interconnects per node. Unfortunately, state-of-the-art production quality implementations of the popular Message Passing Interface (MPI) programming model do not have the appropriate support to deliver the best performance and scalability for applications on such dense GPU systems. These developments in High-End Computing (HEC) technologies and associated middleware issues lead to the following broad challenge: How can existing production quality MPI middleware be enhanced to take advantage of emerging networking technologies to deliver the best possible scale-up and scale-out for HPC and Deep Learning (DL) applications on emerging dense GPU systems? A synergistic and comprehensive research plan, involving computer scientists from The Ohio State University (OSU) and Ohio Supercomputer Center (OSC) and computational scientists from the Texas Advanced Computing Center (TACC), and San Diego Supercomputer Center (SDSC) and University of California San Diego (UCSD), is proposed to address the above broad challenges with innovative solutions. The proposed framework will be made available to collaborators and the broader scientific community to understand the impact of the proposed innovations on next-generation HPC and DL frameworks and applications in various science domains. Multiple graduate and undergraduate students will be trained under this project as future scientists and engineers in HPC. The proposed work will enable curriculum advancements via research in pedagogy for key courses in the new Data Science programs at OSU, SDSC and TACC. The established national-scale training and outreach programs at TACC, SDSC and OSC will be used to disseminate the results of this research to XSEDE users. Tutorials and workshops will be organized at PEARC, SC and other conferences to share the research results and experience with the community. The project is aligned with the National Strategic Computing Initiative (NSCI) to advance US leadership in HPC and the recent initiative of the US Government to maintain leadership in Artificial Intelligence (AI.)The proposed innovations include: 1) Designing high-performance and scalable point-to-point, and collective communication operations that fully utilize multiple network adapters and advanced in-network computing features for GPU and CPU buffers within and across nodes; 2) Designing novel datatype processing and unified memory management to improve application performance; 3) Designing CUDA-aware I/O subsystem to accelerate MPI I/O and checkpoint-restart for HPC and DL applications; 4) Designing support for containerized environments to better enable easy deployment of proposed solutions on modern cloud environments; and 5) Carry out integrated development and evaluation to ensure proper integration of proposed designs with the driving applications. The proposed designs will be integrated into the widely-used MVAPICH2 library and made available. The project team members will work closely with internal and external collaborators to facilitate wide deployment and adoption of released software. The proposed solutions will be targeted to enable scale-up and scale-out of the driving science domains (molecular dynamics, lattice QCD, seismology, image classification, and fusion research) on emerging dense GPU platforms. The transformative impact of the proposed development effort is to achieve scalability, performance, and portability out of HPC and DL frameworks and applications to take advantage of emerging dense GPU platforms and hence, leading to significant advancements in science and engineering.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.