Collaborative Research: EMBRACE: Evolvable Methods for Benchmarking Realism through Application and Community Engagement

高性能计算在科学、规划、健康和国防方面开辟了新的途径，这些都是纯实验无法在经济上实现的。新的进展领域需要新的评估方法，这些方法包含在基准中，用于选择计算硬件和系统。拥抱将指导针对不同科学领域不同需求的计算技术的选择。目前的方法假设一种平台可以解决所有问题，但在生物信息学、城市规划、医学和更多领域的日益广泛的使用表明了新的、未涵盖的系统需求。拥抱通过社区比赛开启了这一过程。这些竞赛既将为多个科学领域建立有用的基准，也将围绕基准科学建立一个社区。不断发展的测量将帮助科学家将他们的应用程序与计算平台最佳匹配。尽管基准测试长期以来一直很有兴趣和重要影响，但对于设计、开发和发展前瞻性基准以及正确解释他们的结果的最合理方式还没有解决方案。随着时间的推移，社区扩大了，平台发生了戏剧性的变化，应用程序也发生了变化，但今天的基准并不一定与它们一起发展。这项建议的主要目标是根据当前和未来的需要，可持续地推进基准科学。第二个目标是创造可持续的激励机制，推动研究和创新。第三个目标是产生一组最有用和不断发展的基准、基准方法或基准分析，以供社区接受。这些愿景和目标将通过建立一个新的、可持续的和社区驱动的竞赛--通过应用和社区参与来确定现实主义基准的可演变方法(拥抱)研讨会来实现，在研讨会上，团队在与绩效衡量和基准直接相关的类别中进行竞争。参赛作品包括代码、技术论文和演示文稿。在与不同应用领域的领导和供应商代表协商后，该团队将负责确定竞赛类别、制定规则、招募参赛者和评委，并在进行过程中进行监测和管理。它将是一个开放的论坛，以促进更广泛的科学界对基准领域基本问题的理解。

High-performance computing opens new avenues in science, planning, health, and defense that are not economically achievable by pure experimentation. New progress areas require new evaluation methods, packaged in benchmarks, for selecting computational hardware and systems. EMBRACE will guide selection of computing technologies for different scientific areas with different needs. Current approaches assume that one kind of platform addresses all problems, but growing use in bioinformatics, urban planning, medicine, and more fields have demonstrated new, uncovered system requirements. EMBRACE opens the process through community competitions. These competitions will both build benchmarks useful for multiple scientific areas as well as build a community around benchmarking science. The evolving measurements will help scientists best match their applications with computing platforms.Despite the long-standing interest and important impacts of benchmarking, there is no resolution on the most rational way to design, develop, and evolve forward-looking benchmarks, and to correctly interpret their results. Over time, the community has broadened, the platforms have changed dramatically, and the applications have evolved, yet today's benchmarks have not necessarily evolved with them. The primary objective of this proposal is to sustainably advance the science of benchmarking based on current and future needs. A second objective is to create sustainable incentives that drive research and innovation. A third objective is to produce a select set of the most useful and evolving benchmarks, benchmarking methods, or benchmarking analyses that can be accepted by the community. The vision and objectives will be realized by establishing a new, sustainable, and community-driven competition, the Evolvable Methods for Benchmarking Realism through Application and Community Engagement (EMBRACE) Workshop, in which teams compete in categories directly relevant to performance measurement and benchmarking. Submissions to the contest will include code, technical papers and presentations. In consultation with leaders in various application domains and with vendor representatives, the EMBRACE team will be responsible for defining the contest categories, framing the rules, recruiting both the participants and the judges, and monitoring and managing the process as it takes place. It will be an open forum to advance the broader scientific community's understanding of fundamental questions in the area of benchmarking.