SHF: Large: Collaborative Research: Next Generation Communication Mechanisms exploiting Heterogeneity, Hierarchy and Concurrency for Emerging HPC Systems

该奖项得到了CIF21软件重用项目的部分支持，该项目的目标是支持通过重用实现可持续软件元素的途径，并强调可重用软件元素在可持续软件网络基础设施中的关键作用，以支持计算和数据支持的科学和工程。基于MPI（消息传递接口）的并行编程在学术界、政府（国防和非国防用途）以及可扩展机器学习和大数据分析中的新兴用途中得到越来越多的应用。密集多核（DMC）架构的出现，如英特尔的Knights Landing （KNL）和加速器/协处理器架构，如NVIDIA的gpgpu，使设计具有高计算密度的系统成为可能。这一点，再加上支持远程直接内存访问（RDMA）的商用网络技术的可用性，如InfiniBand、RoCE和带有iWARP的10/40GigE，正在推动千万亿次和ExaFlop系统的增长。这些DMC架构具有以下独特的特征：层次存储器的更深层次；革命性的网络互联；异构的计算能力和数据移动成本（在芯片级和节点级具有异构性）。对于这些新兴系统，MPI和其他编程模型的组合，称为MPI+X（其中X可以是PGAS， Tasks, OpenMP， OpenACC或CUDA），正在成为目标。当前一代的MPI+X编程模型的通信协议和机制不能有效地支持新兴的DMC体系结构。这导致了以下广泛的挑战：1)如何为下一代DMC架构设计高性能和可扩展的通信机制以支持MPI+X（包括基于任务的）编程模型？2)当前和下一代应用程序如何设计/协同设计所提出的通信机制？俄亥俄州立大学（OSU）和俄亥俄超级计算机中心（OSC）的计算机科学家以及德克萨斯高级计算中心（TACC）、圣地亚哥超级计算机中心（SDSC）和加州大学圣地亚哥分校（UCSD）的计算科学家提出了一项协同综合研究计划，以创新的解决方案解决上述广泛的挑战。该研究将由NSF计算科学研究人员在OSC和OSU的Stampede和Comet以及其他系统上运行大规模模拟的一系列应用程序驱动。建议的设计将集成到广泛使用的MVAPICH2库中，并可供公众使用。该项目将培养多名研究生和本科生成为高性能计算领域的未来科学家和工程师。TACC、SDSC和OSC已建立的全国性培训和推广方案将用于向XSEDE用户传播这项研究的结果。将在XSEDE、SC和其他会议上组织教程，与社区分享研究成果和经验。

This award was partially supported by the CIF21 Software Reuse Venture whose goals are to support pathways towards sustainable software elements through their reuse, and to emphasize the critical role of reusable software elements in a sustainable software cyberinfrastructure to support computational and data-enabled science and engineering.Parallel programming based on MPI (Message Passing Interface) is being used with increased frequency in academia, government (defense and non-defense uses), as well as emerging uses in scalable machine learning and big data analytics. The emergence of Dense Many-Core (DMC) architectures like Intel's Knights Landing (KNL) and accelerator/co-processor architectures like NVIDIA GPGPUs are enabling the design of systems with high compute density. This, coupled with the availability of Remote Direct Memory Access (RDMA)-enabled commodity networking technologies like InfiniBand, RoCE, and 10/40GigE with iWARP, is fueling the growth of multi-petaflop and ExaFlop systems. These DMC architectures have the following unique characteristics: deeper levels of hierarchical memory; revolutionary network interconnects; and heterogeneous compute power and data movement costs (with heterogeneity at chip-level and node-level). For these emerging systems, a combination of MPI and other programming models, known as MPI+X (where X can be PGAS, Tasks, OpenMP, OpenACC, or CUDA), are being targeted. The current generation communication protocols and mechanisms for MPI+X programming models cannot efficiently support the emerging DMC architectures. This leads to the following broad challenges: 1) How can high-performance and scalable communication mechanisms for next generation DMC architectures be designed to support MPI+X (including Task-based) programming models? and 2) How can the current and next generation applications be designed/co-designed with the proposed communication mechanisms?A synergistic and comprehensive research plan, involving computer scientists from The Ohio State University (OSU) and Ohio Supercomputer Center (OSC) and computational scientists from the Texas Advanced Computing Center (TACC), San Diego Supercomputer Center (SDSC) and University of California San Diego (UCSD), is proposed to address the above broad challenges with innovative solutions. The research will be driven by a set of applications from established NSF computational science researchers running large scale simulations on Stampede and Comet and other systems at OSC and OSU. The proposed designs will be integrated into the widely-used MVAPICH2 library and made available for public use. Multiple graduate and undergraduate students will be trained under this project as future scientists and engineers in HPC. The established national-scale training and outreach programs at TACC, SDSC and OSC will be used to disseminate the results of this research to XSEDE users. Tutorials will be organized at XSEDE, SC and other conferences to share the research results and experience with the community.