Collaborative Research: SI2-SSI: A Comprehensive Performance Tuning Framework for the MPI Stack

消息传递接口(MPI)是现代高端计算(HEC)系统上广泛使用的并行编程模型。MPI库的许多性能方面，如延迟、带宽、可伸缩性、内存占用、缓存污染、计算和通信重叠等，都高度依赖于系统配置和应用程序要求。此外，随着多核处理器和商用网络技术(如InfiniBand和10GigE/iWARP)的发展，现代群集正在快速变化。它们正变得多样化和异构化，具有不同数量的处理器内核、处理器速度、内存速度、多代网络适配器/交换机、I/O接口技术和加速器(GPGPU)等。通常，任何MPI库都通过使用各种运行时参数来处理平台中的上述种类的多样性和应用程序的敏感性。这些参数在其发布期间进行调整，或由系统管理员或最终用户进行调整。这些默认参数对于所有系统配置和应用程序可能是最优的，也可能不是最优的。典型专有系统的MPI库针对一系列应用程序经历了大量的性能调整。由于商用集群在其配置(处理器、内存和网络)方面提供了更大的灵活性，因此使用任何MPI库的发布版本(其默认设置)很难实现最佳调整。这带来了以下广泛的挑战：“能否为MPI库设计一个全面的性能调优框架，以便下一代InfiniBand、10GigE/iWARP和ROCE集群和应用程序能够提取‘裸机’的性能和最大的可扩展性？”调查人员包括来自俄亥俄州立大学(OSU)和俄亥俄超级计算机中心(OSC)的计算机科学家，以及来自德克萨斯州高级计算中心(TACC)和圣地亚哥超级计算机中心(SDSC)、加州大学圣地亚哥分校(UCSD)的计算科学家，调查人员将具体解决以下挑战：1)能否设计一套静态工具来优化MPI库在安装期间的性能？2)能否设计一套低开销的动态工具，以优化生产运行期间每个用户和每个应用程序的性能？3)如何将拟议的性能调整框架与即将推出的MPIT接口结合起来？4)如何在给定系统上配置MPI库，以便为一组驱动应用程序提供不同的优化？5)在性能、可伸缩性、内存效率和减少缓存污染方面，建议的调优框架可以实现哪些好处？这项研究将由一组来自NSF计算科学研究人员的应用程序推动，这些研究人员在TACC Ranger和OSC、SDSC和OSU的其他系统上进行大规模模拟。建议的设计将被集成到开源的MVAPICH2库中。

The Message Passing Interface (MPI) is a very widely used parallel programming model on modern High-End Computing (HEC) systems. Many performance aspects of MPI libraries, such as latency, bandwidth, scalability, memory footprint, cache pollution, overlap of computation and communication etc. are highly dependent on system configuration and application requirements. Additionally, modern clusters are changing rapidly with the growth of multi-core processors and commodity networking technologies such as InfiniBand and 10GigE/iWARP. They are becoming diverse and heterogeneous with varying number of processor cores, processor speed, memory speed, multi-generation network adapters/switches, I/O interface technologies, and accelerators (GPGPUs), etc. Typically, any MPI library deals with the above kind of diversity in platforms and sensitivity of applications by employing various runtime parameters. These parameters are tuned during its release, or bysystem administrators, or by end-users. These default parameters may or may not be optimal for all system configurations and applications.The MPI library of a typical proprietary system goes through heavy performance tuning for a range of applications. Since commodity clusters provide greater flexibility in their configurations (processor, memory and network), it is very hard to achieve optimal tuning using released version of any MPI library, with its default settings. This leads to the following broad challenge: "Can a comprehensive performance tuning framework be designed for MPI library so that the next generation InfiniBand, 10GigE/iWARP and RoCE clusters and applications will be able to extract `bare-metal' performance and maximum scalability?" The investigators, involving computerscientists from The Ohio State University (OSU) and Ohio Supercomputer Center (OSC) as well as computational scientists from the Texas Advanced Computing Center (TACC) and San Diego Supercomputer Center (SDSC), University of California San Diego (UCSD), will be addressing the above challenge with innovative solutions.The investigators will specifically address the following challenges: 1) Can a set of static tools be designed to optimize performance of an MPI library during installation time? 2) Can a set of dynamic tools with low overhead be designed to optimize performance on a per-user and per-application basis during production runs? 3) How to incorporate the proposed performance tuning framework with the upcoming MPIT interface? 4) How to configure MPI libraries on a given system to deliver different optimizations to a set of driving applications? and 5) What kind of benefits (in terms of performance, scalability, memory efficiency and reduction in cache pollution) can be achieved by the proposed tuning framework? The research will be driven by a set of applications from established NSF computational science researchers running large scale simulations on the TACC Ranger and other systems at OSC, SDSC and OSU. The proposed designs will be integrated into the open-source MVAPICH2 library.