Collaborative Research: EMBRACE: Evolvable Methods for Benchmarking Realism through Application and Community Engagement

高性能计算在科学、规划、健康和国防领域开辟了新的途径，这些途径在经济上是无法通过纯粹的实验实现的。新的进步领域需要新的评估方法，包装在基准中，用于选择计算硬件和系统。EMBRACE将指导针对不同科学领域的不同需求选择计算技术。目前的方法假设一种平台可以解决所有问题，但是在生物信息学、城市规划、医学和更多领域的日益增长的应用已经证明了新的、未发现的系统需求。EMBRACE通过社区竞赛开启了这一进程。这些竞赛既将为多个科学领域建立有用的基准，也将围绕科学基准建立一个社区。不断发展的测量方法将帮助科学家最好地将他们的应用与计算平台相匹配。尽管对基准的长期关注和重要影响，但对于设计、开发和发展前瞻性基准以及正确解释其结果的最合理方式尚无定论。随着时间的推移，社区扩大了，平台发生了巨大的变化，应用程序也在发展，但今天的基准并不一定会随之发展。本提案的主要目标是基于当前和未来的需求，可持续地推进基准科学。第二个目标是建立可持续的激励机制，推动研究和创新。第三个目标是产生一组可被社区接受的最有用和不断发展的基准、基准测试方法或基准测试分析。愿景和目标将通过建立一个新的、可持续的、社区驱动的竞赛来实现，即通过应用和社区参与实现基准现实的可进化方法（EMBRACE）研讨会，在该研讨会中，团队在与绩效衡量和基准直接相关的类别中进行竞争。参赛作品将包括代码、技术论文和演示文稿。在与各个应用领域的领导者和供应商代表协商后，EMBRACE团队将负责定义竞赛类别，制定规则，招募参与者和评委，并监控和管理整个过程。这将是一个开放的论坛，以促进更广泛的科学界对基准领域基本问题的理解。

High-performance computing opens new avenues in science, planning, health, and defense that are not economically achievable by pure experimentation. New progress areas require new evaluation methods, packaged in benchmarks, for selecting computational hardware and systems. EMBRACE will guide selection of computing technologies for different scientific areas with different needs. Current approaches assume that one kind of platform addresses all problems, but growing use in bioinformatics, urban planning, medicine, and more fields have demonstrated new, uncovered system requirements. EMBRACE opens the process through community competitions. These competitions will both build benchmarks useful for multiple scientific areas as well as build a community around benchmarking science. The evolving measurements will help scientists best match their applications with computing platforms.Despite the long-standing interest and important impacts of benchmarking, there is no resolution on the most rational way to design, develop, and evolve forward-looking benchmarks, and to correctly interpret their results. Over time, the community has broadened, the platforms have changed dramatically, and the applications have evolved, yet today's benchmarks have not necessarily evolved with them. The primary objective of this proposal is to sustainably advance the science of benchmarking based on current and future needs. A second objective is to create sustainable incentives that drive research and innovation. A third objective is to produce a select set of the most useful and evolving benchmarks, benchmarking methods, or benchmarking analyses that can be accepted by the community. The vision and objectives will be realized by establishing a new, sustainable, and community-driven competition, the Evolvable Methods for Benchmarking Realism through Application and Community Engagement (EMBRACE) Workshop, in which teams compete in categories directly relevant to performance measurement and benchmarking. Submissions to the contest will include code, technical papers and presentations. In consultation with leaders in various application domains and with vendor representatives, the EMBRACE team will be responsible for defining the contest categories, framing the rules, recruiting both the participants and the judges, and monitoring and managing the process as it takes place. It will be an open forum to advance the broader scientific community's understanding of fundamental questions in the area of benchmarking.