Collaborative Research: EMBRACE: Evolvable Methods for Benchmarking Realism through Application and Community Engagement

高性能计算在科学、规划、健康和国防领域开辟了新的途径，而这些都是纯实验在经济上无法实现的。新的进展领域需要新的评估方法，包装在基准，选择计算硬件和系统。EMBRACE将为具有不同需求的不同科学领域选择计算技术提供指导。目前的方法假设一种平台可以解决所有问题，但在生物信息学、城市规划、医学和更多领域的日益增长的使用已经证明了新的、未被覆盖的系统需求。EMBRACE通过社区竞赛开启这一进程。这些竞赛将建立对多个科学领域有用的基准，并围绕基准科学建立一个社区。不断发展的测量将帮助科学家最好地匹配他们的应用程序与计算平台。尽管基准测试的长期兴趣和重要影响，没有解决方案的最合理的方式来设计，开发和发展前瞻性基准测试，并正确解释他们的结果。随着时间的推移，社区已经扩大，平台已经发生了巨大的变化，应用程序已经发展，但今天的基准测试并不一定与它们一起发展。本提案的主要目标是根据当前和未来的需要，可持续地推进基准科学。第二个目标是建立可持续的激励机制，推动研究和创新。第三个目标是产生一套最有用和不断发展的基准、基准方法或基准分析，这些基准、基准方法或基准分析可以被社区接受。愿景和目标将通过建立一个新的，可持续的，社区驱动的竞争来实现，通过应用和社区参与（EMBRACE）研讨会对现实主义进行基准测试的可发展方法，在该研讨会上，团队在与绩效衡量和基准测试直接相关的类别中进行竞争。参赛作品将包括代码、技术论文和演示文稿。在与各个应用领域的领导者和供应商代表协商后，EMBRACE团队将负责定义竞赛类别，制定规则，招募参赛者和评委，并在比赛过程中进行监控和管理。这将是一个公开的论坛，以促进更广泛的科学界对基准领域基本问题的理解。

High-performance computing opens new avenues in science, planning, health, and defense that are not economically achievable by pure experimentation. New progress areas require new evaluation methods, packaged in benchmarks, for selecting computational hardware and systems. EMBRACE will guide selection of computing technologies for different scientific areas with different needs. Current approaches assume that one kind of platform addresses all problems, but growing use in bioinformatics, urban planning, medicine, and more fields have demonstrated new, uncovered system requirements. EMBRACE opens the process through community competitions. These competitions will both build benchmarks useful for multiple scientific areas as well as build a community around benchmarking science. The evolving measurements will help scientists best match their applications with computing platforms.Despite the long-standing interest and important impacts of benchmarking, there is no resolution on the most rational way to design, develop, and evolve forward-looking benchmarks, and to correctly interpret their results. Over time, the community has broadened, the platforms have changed dramatically, and the applications have evolved, yet today's benchmarks have not necessarily evolved with them. The primary objective of this proposal is to sustainably advance the science of benchmarking based on current and future needs. A second objective is to create sustainable incentives that drive research and innovation. A third objective is to produce a select set of the most useful and evolving benchmarks, benchmarking methods, or benchmarking analyses that can be accepted by the community. The vision and objectives will be realized by establishing a new, sustainable, and community-driven competition, the Evolvable Methods for Benchmarking Realism through Application and Community Engagement (EMBRACE) Workshop, in which teams compete in categories directly relevant to performance measurement and benchmarking. Submissions to the contest will include code, technical papers and presentations. In consultation with leaders in various application domains and with vendor representatives, the EMBRACE team will be responsible for defining the contest categories, framing the rules, recruiting both the participants and the judges, and monitoring and managing the process as it takes place. It will be an open forum to advance the broader scientific community's understanding of fundamental questions in the area of benchmarking.