EAGER: Measuring Real World Application Performance on Next-Generation Computing Systems

该项目建议使用现实世界的科学应用程序创建一个基准套件。基准套件将与标准性能评估公司（SPEC）的高性能组（HPG）共同开发，标准性能评估公司是一个非营利组织，其目标是创建和维护标准化基准，以评估最新一代计算系统的性能和能源效率。该项目将利用SPEC基础设施来确保基准套件的持续维护，以及对结果提交、审查和在SPEC网站上发布的支持。此外，该项目将使用基准测试套件的公布结果来创建全球部署的高性能计算（HPC）系统的公开排名。与SPEC的伙伴关系将确保项目的成果将传播到工业界。与SPEC/HPG共同创建一个新的真实世界的应用程序基准套件，以及适合应用程序基准的性能指标的开发，将导致改进HPC系统设计，并使人们更好地理解下一代计算系统必须如何发展。除了对高性能计算系统进行评估外，拟议的应用基准套件还将对新型软件和硬件生态系统进行比较性能评估。此外，建议的工作建立在NSF在领域科学应用程序开发方面的投资上，通过将这些应用程序包括在套件中。重要的是，该项目将是一个机会，将基准社区聚集在一起，为现实世界的科学应用开发和定义性能和吞吐量指标。最后，该项目将提供一个基准套件，该套件将免费提供给非营利组织，并由spec提供支持。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，认为值得支持。

The project proposes to create a benchmark suite using real-world scientific applications. The benchmark suite will be developed jointly with the High-Performance Group (HPG) of the Standard Performance Evaluation Corporation (SPEC), a non-profit organization with the goal of creating and maintaining standardized benchmarks to evaluate performance and energy efficiency for the newest generation of computing systems. The project will leverage SPEC infrastructure to ensure continuous maintenance of the benchmark suite, as well as support for result submission, review, and publication on the SPEC website. Furthermore, the project will use the published results of the benchmark suite to create a public ranking of High-Performance Computing (HPC) systems deployed across the world. Partnership with SPEC will ensure that the results of the project will be disseminated to industry.The creation of a new real-world application benchmark suite jointly with SPEC/HPG, as well as the development of performance metrics suitable for application benchmarks, will lead to improved HPC system design and enable the better understanding of how next-generation computing systems must evolve. In addition to evaluating HPC systems, the proposed application benchmark suite will also enable the comparative performance evaluation of novel software and hardware ecosystems. Moreover, the proposed work builds on NSF investment in domain science application development by including these applications in the suite. Importantly, the project will be an opportunity to bring the benchmark community together to develop and define performance and throughput metrics for real world scientific applications. Finally, the project will deliver a benchmark suite that will be freely available to non-profit organizations and sustained by SPEC.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.