CSR: Rethinking System Software for Overprovisioned, High-Performance Computing Systems

目前，高性能计算（HPC）社区专注于实现exaflop性能，这是当今世界上最好的超级计算机性能的30倍。出于实际、财务和环境方面的考虑，能源部将实现艾弗洛普的功率限制在20兆瓦。 由于当今的顶级机器通常消耗5到20兆瓦的电力，但距离exaflop性能目标还有一个数量级或更多，因此HPC系统的硬件和软件必须取得重大进展。 改进硬件的一种方法是使用过度配置的系统，这些系统包含的机器数量超过了可以同时完全供电的数量。 虽然过度供应的系统有可能大大提高功率和性能，但需要重新设计软件以支持此类系统。本提案的重点是设计和实施支持过度供应系统的软件基础设施。 基础设施的关键进步是支持系统范围的优化，即，跨多个应用程序的优化。这与当前HPC系统中基于每个应用程序进行优化的重点形成鲜明对比。 开发的软件将包括一个作业分析器，一个一次对多个作业执行分析的调度器，以及一个基于调度器分析的输出联合优化多个应用程序的集群范围的运行时系统。实现exascale计算是一个重要的国家优先事项，将影响许多关键应用领域，如气候/天气，可再生能源，核能，材料科学，和国家安全 这里描述的工作将提高功耗受限的HPC系统的整体系统性能，这是迈向exascale目标的重要一步。 该项目计划通过与几个国家实验室的长期合作，以拟议软件栈的形式转让这项研究所产生的技术。

Currently, the high-performance computing (HPC) community is focused on achieving exaflop performance, which is about a 30-fold improvement from the performance of the best supercomputer in the world today. Because of practical, financial, and environmental concerns, the Department of Energy is setting a power limit for achieving an exaflop at 20 megawatts. As today's top machines generally consume between five and 20 megawatts---and yet are an order of magnitude or more away from the exaflop performance target, significant hardware and software advances in HPC systems are necessary. One way to improve hardware is to use overprovisioned systems, which contain more machines than can be fully powered simultaneously. While overprovisioned systems have the potential to significantly improve power and performance, software will need to be redesigned to support such systems.The focus of this proposal is to design and implement software infrastructure that will support overprovisioned systems. The key advance in the infrastructure is support of system-wide optimizations, i.e., optimizations that span multiple applications. This is in stark contrast to the current focus in HPC systems of optimizing on a per-application basis. The developed software will consist of a job profiler, a scheduler that performs analysis on multiple jobs at a time, and a cluster-wide run-time system that jointly optimizes multiple applications based on the output of the scheduler analysis.Achieving exascale computing is an important national priority and will impact many critical application domains, such as climate/weather, renewable energy, nuclear energy, materials science, and national security. The work described here will improve whole-system performance on power-constrained HPC systems, which is one important step towards the exascale goal. The project plans to transfer technology resulting from this research in the form of the proposed software stack via longstanding collaborations with several national laboratories.