Collaborative Research: SI2-SSI: A Comprehensive Performance Tuning Framework for the MPI Stack

消息传递接口（MPI）是现代高端计算（HEC）系统中广泛使用的并行编程模型。MPI库的许多性能方面，如延迟、带宽、可伸缩性、内存占用、缓存污染、计算和通信的重叠等，都高度依赖于系统配置和应用程序需求。此外，随着多核处理器和商用网络技术（如InfiniBand和10GigE/iWARP）的发展，现代集群正在迅速变化。随着处理器核心数量、处理器速度、内存速度、多代网络适配器/交换机、I/O接口技术和加速器（gpgpu）等的变化，它们变得多样化和异构化。通常，任何MPI库都通过使用各种运行时参数来处理平台上的上述多样性和应用程序的敏感性。这些参数在其发布期间或由系统管理员或最终用户进行调优。这些默认参数对于所有系统配置和应用程序可能是最优的，也可能不是。典型专有系统的MPI库针对一系列应用程序进行了大量的性能调优。由于普通集群在配置（处理器、内存和网络）方面提供了更大的灵活性，因此使用任何MPI库的发布版本及其默认设置都很难实现最佳调优。这导致了以下广泛的挑战：“能否为MPI库设计一个全面的性能调优框架，以便下一代InfiniBand、10GigE/iWARP和RoCE集群和应用程序能够提取‘裸机’性能和最大的可扩展性？”研究人员包括来自俄亥俄州立大学（OSU）和俄亥俄超级计算机中心（OSC）的计算机科学家，以及来自德克萨斯高级计算中心（TACC）和圣地亚哥超级计算机中心（SDSC）、加州大学圣地亚哥分校（UCSD）的计算科学家，他们将用创新的解决方案来解决上述挑战。研究人员将具体解决以下挑战：1)能否设计一组静态工具来优化MPI库在安装期间的性能？2)是否可以设计一组低开销的动态工具来优化生产运行期间每个用户和每个应用程序的性能？3)如何将提出的性能调优框架与即将推出的MPIT接口结合起来？4)如何在给定系统上配置MPI库，以便为一组驱动应用程序提供不同的优化？5)提议的调优框架能带来什么样的好处（在性能、可伸缩性、内存效率和减少缓存污染方面）？该研究将由NSF计算科学研究人员在TACC Ranger和OSC、SDSC和OSU的其他系统上运行大规模模拟的一系列应用程序驱动。提出的设计将集成到开源的MVAPICH2库中。

The Message Passing Interface (MPI) is a very widely used parallel programming model on modern High-End Computing (HEC) systems. Many performance aspects of MPI libraries, such as latency, bandwidth, scalability, memory footprint, cache pollution, overlap of computation and communication etc. are highly dependent on system configuration and application requirements. Additionally, modern clusters are changing rapidly with the growth of multi-core processors and commodity networking technologies such as InfiniBand and 10GigE/iWARP. They are becoming diverse and heterogeneous with varying number of processor cores, processor speed, memory speed, multi-generation network adapters/switches, I/O interface technologies, and accelerators (GPGPUs), etc. Typically, any MPI library deals with the above kind of diversity in platforms and sensitivity of applications by employing various runtime parameters. These parameters are tuned during its release, or bysystem administrators, or by end-users. These default parameters may or may not be optimal for all system configurations and applications.The MPI library of a typical proprietary system goes through heavy performance tuning for a range of applications. Since commodity clusters provide greater flexibility in their configurations (processor, memory and network), it is very hard to achieve optimal tuning using released version of any MPI library, with its default settings. This leads to the following broad challenge: "Can a comprehensive performance tuning framework be designed for MPI library so that the next generation InfiniBand, 10GigE/iWARP and RoCE clusters and applications will be able to extract `bare-metal' performance and maximum scalability?" The investigators, involving computerscientists from The Ohio State University (OSU) and Ohio Supercomputer Center (OSC) as well as computational scientists from the Texas Advanced Computing Center (TACC) and San Diego Supercomputer Center (SDSC), University of California San Diego (UCSD), will be addressing the above challenge with innovative solutions.The investigators will specifically address the following challenges: 1) Can a set of static tools be designed to optimize performance of an MPI library during installation time? 2) Can a set of dynamic tools with low overhead be designed to optimize performance on a per-user and per-application basis during production runs? 3) How to incorporate the proposed performance tuning framework with the upcoming MPIT interface? 4) How to configure MPI libraries on a given system to deliver different optimizations to a set of driving applications? and 5) What kind of benefits (in terms of performance, scalability, memory efficiency and reduction in cache pollution) can be achieved by the proposed tuning framework? The research will be driven by a set of applications from established NSF computational science researchers running large scale simulations on the TACC Ranger and other systems at OSC, SDSC and OSU. The proposed designs will be integrated into the open-source MVAPICH2 library.