SHF: Large: Collaborative Research: Next Generation Communication Mechanisms exploiting Heterogeneity, Hierarchy and Concurrency for Emerging HPC Systems

该奖项部分得到了CIF 21软件重用项目的支持，该项目的目标是通过重用支持可持续软件元素的途径，并强调可重用软件元素在可持续软件网络基础设施中的关键作用，以支持计算和数据支持的科学和工程。（消息传递接口）在学术界、政府（国防和非国防用途）以及可扩展机器学习和大数据分析中的新兴用途中的使用频率越来越高。密集众核（DMC）架构（如英特尔的Knights Landing（KNL））和加速器/协处理器架构（如NVIDIA GPGPU）的出现使系统设计具有高计算密度。再加上支持远程直接内存访问（RDMA）的商用网络技术的可用性，如InfiniBand、RoCE和10/40 GigE与iWARP，正在推动多petaflop和ExaFlop系统的增长。这些DMC架构具有以下独特特征：更深层次的分层存储器;革命性的网络互连;以及异构计算能力和数据移动成本（芯片级和节点级的异构性）。对于这些新兴的系统，MPI和其他编程模型的组合，称为MPI+X（其中X可以是PGAS，Tasks，OpenMP，OpenACC或CUDA），正在成为目标。 当前MPI+X编程模型的通信协议和机制不能有效地支持新兴的DMC体系结构。 这导致了以下广泛的挑战：1）如何设计用于下一代DMC架构的高性能和可扩展的通信机制以支持MPI+X（包括基于任务的）编程模型？以及2）如何利用所提出的通信机制来设计/协同设计当前和下一代应用？一个协同和全面的研究计划，涉及计算机科学家从俄亥俄州州立大学（OSU）和俄亥俄州超级计算机中心（OSC）和计算科学家从得克萨斯州高级计算中心（TACC），圣地亚哥超级计算机中心（SDSC）和加州圣地亚哥大学（UCSD），提出了解决上述广泛的挑战与创新的解决方案。 这项研究将由NSF计算科学研究人员在OSC和OSU的Stampede和Comet以及其他系统上进行大规模模拟的一系列应用程序驱动。 拟议的设计将被集成到广泛使用的MVAPICH 2库中，供公众使用。 多个研究生和本科生将在这个项目下培养未来的科学家和工程师在HPC。在TACC、SDSC和OSC建立的全国范围的培训和推广计划将用于向XSEDE用户传播这项研究的结果。学术研讨会将在XSEDE、SC和其他会议上举办，与社区分享研究成果和经验。

This award was partially supported by the CIF21 Software Reuse Venture whose goals are to support pathways towards sustainable software elements through their reuse, and to emphasize the critical role of reusable software elements in a sustainable software cyberinfrastructure to support computational and data-enabled science and engineering.Parallel programming based on MPI (Message Passing Interface) is being used with increased frequency in academia, government (defense and non-defense uses), as well as emerging uses in scalable machine learning and big data analytics. The emergence of Dense Many-Core (DMC) architectures like Intel's Knights Landing (KNL) and accelerator/co-processor architectures like NVIDIA GPGPUs are enabling the design of systems with high compute density. This, coupled with the availability of Remote Direct Memory Access (RDMA)-enabled commodity networking technologies like InfiniBand, RoCE, and 10/40GigE with iWARP, is fueling the growth of multi-petaflop and ExaFlop systems. These DMC architectures have the following unique characteristics: deeper levels of hierarchical memory; revolutionary network interconnects; and heterogeneous compute power and data movement costs (with heterogeneity at chip-level and node-level). For these emerging systems, a combination of MPI and other programming models, known as MPI+X (where X can be PGAS, Tasks, OpenMP, OpenACC, or CUDA), are being targeted. The current generation communication protocols and mechanisms for MPI+X programming models cannot efficiently support the emerging DMC architectures. This leads to the following broad challenges: 1) How can high-performance and scalable communication mechanisms for next generation DMC architectures be designed to support MPI+X (including Task-based) programming models? and 2) How can the current and next generation applications be designed/co-designed with the proposed communication mechanisms?A synergistic and comprehensive research plan, involving computer scientists from The Ohio State University (OSU) and Ohio Supercomputer Center (OSC) and computational scientists from the Texas Advanced Computing Center (TACC), San Diego Supercomputer Center (SDSC) and University of California San Diego (UCSD), is proposed to address the above broad challenges with innovative solutions. The research will be driven by a set of applications from established NSF computational science researchers running large scale simulations on Stampede and Comet and other systems at OSC and OSU. The proposed designs will be integrated into the widely-used MVAPICH2 library and made available for public use. Multiple graduate and undergraduate students will be trained under this project as future scientists and engineers in HPC. The established national-scale training and outreach programs at TACC, SDSC and OSC will be used to disseminate the results of this research to XSEDE users. Tutorials will be organized at XSEDE, SC and other conferences to share the research results and experience with the community.