SDCI HPC Improvement: IPM- A Performance Monitoring Environment for Petascale High-Performance Computing Sytems

该项目的目标是通过一个轻量级的性能分析工具来提高NSF中心和其他地方的高性能计算机器的利用率，该工具可以在生产运行期间识别全面应用程序的性能瓶颈。NSF在一级和二级计算机上的投资，以及在大学和工业环境中越来越受欢迎的小型集群，为使用计算科学的新科学发现提供了巨大的机会。然而，经验表明，许多用户并没有有效地利用这些机器，他们往往依赖于算法、编程工具或库，而这些算法、编程工具或库会遇到可消除的性能瓶颈。高并发性、复杂的处理器架构、脆弱的编译器优化、低度网络拓扑、深度内存层次结构、负载不平衡以及由于操作系统噪声而导致的不可预测的性能，这些架构特征使得性能瓶颈既容易遇到又难以发现。我们建议进行研究和开发，为用户和系统管理员提供一种工具，用于识别生产中的性能瓶颈。我们将扩展和部署集成性能监控（IPM）工具，用于识别通信瓶颈、内存系统瓶颈、负载不平衡以及从小型集群到千万亿级系统上的其他性能问题。我们将IPM开发为一个超轻量级的性能分析系统。目前的版本在NSF、DOE和DOD高性能计算中心使用。IPM具有独特的特性，使其能够有效地用于系统管理员和应用程序科学家对应用程序性能的持续监视。IPM的主要特点包括：一个高度可扩展的性能分析策略，对性能的干扰小于5%；与性能数据库集成，允许在应用程序运行和用户之间进行简单和即时的比较；并且易于重新编译。通过进一步的开发，我们将提供：1)一个捕获程序性能数据的工具，特别强调低开销和高达数百万处理器的可扩展性；2)易于理解的应用程序配置文件，它捕获通信量和模式，处理器和内存系统计数器信息，以及来自网络适配器和交换机的拓扑感知计数器3)用于工作负载表征和架构分析的数据库后端；4)通过我们的可移植、可扩展、开源软件支持社区驱动的增强功能。智力优势：我们将通过使IPM在更多更大的机器上运行来扩展IPM的广度，并包含其他重要的性能信息。因此，我们将使领域科学家能够查明影响其应用程序在具有深度内存层次结构、复杂网络拓扑结构和分层并行性的机器上运行的性能问题。我们将帮助科学家快速回答诸如，影响我的科学应用程序性能的因素是什么？”此外，我们的基础设施将回答关于架构特性的好处的基本问题，例如单侧通信、高度网络、内存系统结构和处理器加速器。它还将支持跨千万亿级系统的应用程序性能分析，自动和手动性能调优，以及使用完整机器和真实输入数据的算法可伸缩性的“原位”分析。更广泛的影响：通过我们的可伸缩、可移植和可扩展的方法，我们将以低开销为性能分析带来透明度。我们将使所有HPC利益相关者能够评估和改进应用程序和架构，教育用户有关性能特征，并确保并行机器被有效地用于回答科学和工程中的基本问题。此外，通过在领域科学家、NSF中心和HPC供应商之间提供密切的工作关系，该项目将教育那些在影响HPC软件和硬件设计的许多方面受过培训的学生。一个副产品将增加对如何最佳地使用当前和即将到来的NSF高性能计算Tier1和Tier2系统组合的理解。

Project SummaryThe goal of this project is to improve the utilization of HPC machines at NSF centers and elsewhere via a lightweight performance profiling tool that can identify performance bottlenecks in full scale applications during production runs. Investments by NSF in Tier1 and Tier2 computers, as well as ever-growing popularity of smaller clusters in university and industrial settings, offer tremendous opportunities for new scientific discoveries using computational science. Yet experience suggests that many users do not make effective use of these machines, often relying on algorithms, programming tools, or libraries that encounter removable performance bottlenecks. High concurrency, complex processor architectures, fragile compiler optimizations, low degree network topologies, deep memory hierarchies, load imbalance, and unpredictable performance due to OS noise are among architectural features that make performance bottlenecks simultaneously easy to encounter and hard to find.We propose research and development to provide users and system administrators with a tool for identifying performance bottlenecks in production. We will extend and deploy our Integrated Performance Monitoring (IPM) tool for identifying communication bottlenecks, memory system bottlenecks, load imbalances, and other performance problems on systems ranging from small clusters to the petascale. We developed IPM as an ultra lightweight performance profiling system. The current version is in use at NSF, DOE, and DOD HPC centers. IPM has unique features that make it effective for ongoing monitoring of application performance by system administrators as well as application scientists. The key features of IPM include: a performance profiling strategy that is highly scalable and perturbs performance by less that 5%; integration with a performance database that allows for easy and immediate comparisons across applications runs and users; and an easy to recompilation. Via further development we will provide: 1) A tool for capturing program's performance data with special emphasis on low overhead and scalability for up to millions of processors; 2) Easy to understand application profiles which capture communication volumes and patterns, processor and memory system counter information, and topology-aware counters from network adapters and switches 3) A database backend for workload characterization and architecture analytics; 4) Support for community driven enhancements through our portable, extensible, Open Source software.Intellectual Merit: We will extend IPM's breadth by making it run on more and larger machines, and include additional important performance information. Thereby we will enable domain scientists to pinpoint performance issues affecting their applications running on machines with deep memory hierarchies, complex network topologies, and hierarchical parallelism. We will help scientists to quickly answer questions such as, what are the factors affecting the performance of my scientific application?" In addition, our infrastructure will answer fundamental questions about the benefits of architectural features, such as one sided communication, high degree networks, memory system structures, and processor accelerators. It will also support application performance analysis across petascale systems, automatic and manual performance tuning, and in situ" analysis of algorithm scalability using a full machine and real input data.Broader Impact: Through our scalable, portable, and extensible approach we will bring transparency to performance analysis with low overhead. We will enable all HPC stakeholders to assess and improve both applications and architectures, educate users on performance features, and ensure that parallel machines are used productively to answer basic questions in science and engineering. In addition, by providing a close working relationship between domain scientists, NSF centers, and HPC vendors, this project will educate students who are trained in the many facets that impact HPC software and hardware design. A byproduct will be increased understanding of how to optimally use the current and upcoming NSF HPC Tier1 and Tier2 systems portfolio.