SDCI HPC Improvement: IPM- A Performance Monitoring Environment for Petascale High-Performance Computing Sytems

项目摘要本项目的目标是通过轻量级性能分析工具提高NSF中心和其他地方的HPC机器的利用率，该工具可以在生产运行期间识别全面应用程序中的性能瓶颈。NSF对Tier1和Tier2计算机的投资，以及大学和工业环境中越来越受欢迎的较小群集，为使用计算科学进行新的科学发现提供了巨大的机会。然而，经验表明，许多用户没有有效地使用这些机器，通常依赖于遇到可移除的性能瓶颈的算法、编程工具或库。高并发性、复杂的处理器体系结构、脆弱的编译器优化、低度网络拓扑、深度内存层次结构、负载不平衡以及操作系统噪声导致的不可预测的性能等体系结构特征使得性能瓶颈既容易遇到又难以发现。我们建议进行研究和开发，为用户和系统管理员提供一种工具来识别生产中的性能瓶颈。我们将扩展和部署我们的集成性能监控(IPM)工具，以识别通信瓶颈、内存系统瓶颈、负载不平衡和其他性能问题，范围从小型集群到千万亿级的系统。我们开发的IPM是一个超轻量级的性能分析系统。当前版本正在NSF、DOE和DOD HPC中心使用。IPM具有独特的功能，使其能够有效地由系统管理员和应用程序科学家持续监控应用程序性能。IPM的主要功能包括：高度可扩展的性能评测策略，对性能的影响低于5%；与性能数据库的集成，允许在应用程序运行和用户之间进行轻松和即时的比较；以及易于重新编译。通过进一步的开发，我们将提供：1)捕获程序性能数据的工具，特别强调低开销和高达数百万处理器的可扩展性；2)易于理解的应用程序配置文件，捕获通信量和模式、处理器和内存系统计数器信息，以及来自网络适配器和交换机的拓扑感知计数器3)用于工作负载表征和体系结构分析的数据库后端；4)通过我们的可移植、可扩展的开源软件支持社区驱动的增强功能。智力优势：我们将通过使IPM在更多更大的计算机上运行来扩展IPM的广度，并包括其他重要的性能信息。因此，我们将使领域科学家能够准确地确定影响其应用程序在具有深度内存层次结构、复杂网络拓扑和层次并行的计算机上运行的性能问题。我们将帮助科学家快速回答以下问题：影响我的科学应用程序性能的因素是什么？“此外，我们的基础设施将回答有关架构功能优势的基本问题，例如单边通信、高度网络、内存系统结构和处理器加速器。它还将支持跨千万亿级系统的应用程序性能分析、自动和手动性能调整，以及使用完整机器和真实输入数据对算法可伸缩性的现场分析。广泛影响：通过我们的可伸缩、可移植和可扩展的方法，我们将以较低的开销为性能分析带来透明度。我们将使所有HPC利益相关者能够评估和改进应用程序和架构，教育用户有关性能特征，并确保并行机器被高效地用于回答科学和工程方面的基本问题。此外，通过在领域科学家、NSF中心和HPC供应商之间提供密切的工作关系，该项目将教育那些在影响HPC软件和硬件设计的许多方面接受培训的学生。副产品将是增加对如何以最佳方式使用当前和即将推出的NSF HPC Tier1和Tier2系统产品组合的理解。

Project SummaryThe goal of this project is to improve the utilization of HPC machines at NSF centers and elsewhere via a lightweight performance profiling tool that can identify performance bottlenecks in full scale applications during production runs. Investments by NSF in Tier1 and Tier2 computers, as well as ever-growing popularity of smaller clusters in university and industrial settings, offer tremendous opportunities for new scientific discoveries using computational science. Yet experience suggests that many users do not make effective use of these machines, often relying on algorithms, programming tools, or libraries that encounter removable performance bottlenecks. High concurrency, complex processor architectures, fragile compiler optimizations, low degree network topologies, deep memory hierarchies, load imbalance, and unpredictable performance due to OS noise are among architectural features that make performance bottlenecks simultaneously easy to encounter and hard to find.We propose research and development to provide users and system administrators with a tool for identifying performance bottlenecks in production. We will extend and deploy our Integrated Performance Monitoring (IPM) tool for identifying communication bottlenecks, memory system bottlenecks, load imbalances, and other performance problems on systems ranging from small clusters to the petascale. We developed IPM as an ultra lightweight performance profiling system. The current version is in use at NSF, DOE, and DOD HPC centers. IPM has unique features that make it effective for ongoing monitoring of application performance by system administrators as well as application scientists. The key features of IPM include: a performance profiling strategy that is highly scalable and perturbs performance by less that 5%; integration with a performance database that allows for easy and immediate comparisons across applications runs and users; and an easy to recompilation. Via further development we will provide: 1) A tool for capturing program's performance data with special emphasis on low overhead and scalability for up to millions of processors; 2) Easy to understand application profiles which capture communication volumes and patterns, processor and memory system counter information, and topology-aware counters from network adapters and switches 3) A database backend for workload characterization and architecture analytics; 4) Support for community driven enhancements through our portable, extensible, Open Source software.Intellectual Merit: We will extend IPM's breadth by making it run on more and larger machines, and include additional important performance information. Thereby we will enable domain scientists to pinpoint performance issues affecting their applications running on machines with deep memory hierarchies, complex network topologies, and hierarchical parallelism. We will help scientists to quickly answer questions such as, what are the factors affecting the performance of my scientific application?" In addition, our infrastructure will answer fundamental questions about the benefits of architectural features, such as one sided communication, high degree networks, memory system structures, and processor accelerators. It will also support application performance analysis across petascale systems, automatic and manual performance tuning, and in situ" analysis of algorithm scalability using a full machine and real input data.Broader Impact: Through our scalable, portable, and extensible approach we will bring transparency to performance analysis with low overhead. We will enable all HPC stakeholders to assess and improve both applications and architectures, educate users on performance features, and ensure that parallel machines are used productively to answer basic questions in science and engineering. In addition, by providing a close working relationship between domain scientists, NSF centers, and HPC vendors, this project will educate students who are trained in the many facets that impact HPC software and hardware design. A byproduct will be increased understanding of how to optimally use the current and upcoming NSF HPC Tier1 and Tier2 systems portfolio.