Reliable benchmarking: requirements and solutions
Reliable benchmarking: requirements and solutions
复制标题
DOI：
10.1007/s10009-017-0469-y
复制
发表时间：
2019-02-01
影响因子：
1.5
通讯作者：
Wendler, Philipp
中科院分区：
文献类型：
--
作者：
Beyer, Dirk;Loewe, Stefan;Wendler, Philipp
基准测试是一种在实验计算机科学中广泛使用的方法，特别是用于工具和算法的比较评估。因此，为了确保适当的基准测试、资源度量和结果的呈现，需要回答许多问题，所有这些对于研究人员、工具开发人员和用户以及工具竞争都是必不可少的。我们确定了一组需求，这些需求对于自动求解器、验证器和类似工具的时间和内存使用的可靠基准测试和资源测量是必不可少的，并讨论了现有方法和基准测试工具的局限性。在基准测试框架中满足这些要求（在Linux系统上）目前只能通过使用内核的cgroup和命名空间特性来完成。我们开发了BenchExec，这是一个现成的、工具独立的、开源的基准测试框架实现，它满足所有提出的需求，使可靠的基准测试和资源测量变得容易。我们的框架能够与各种不同的工具一起工作，在国际软件验证竞赛中证明了它的可靠性和实用性，并被世界各地的几个研究小组使用，以确保可靠的基准测试。最后，我们提出了如何以科学有效和可理解的方式呈现测量结果的指导方针。
Benchmarking is a widely used method in experimental computer science, in particular, for the comparative evaluation of tools and algorithms. As a consequence, a number of questions need to be answered in order to ensure proper benchmarking, resource measurement, and presentation of results, all of which is essential for researchers, tool developers, and users, as well as for tool competitions. We identify a set of requirements that are indispensable for reliable benchmarking and resource measurement of time and memory usage of automatic solvers, verifiers, and similar tools, and discuss limitations of existing methods and benchmarking tools. Fulfilling these requirements in a benchmarking framework can (on Linux systems) currently only be done by using the cgroup and namespace features of the kernel. We developed BenchExec, a ready-to-use, tool-independent, and open-source implementation of a benchmarking framework that fulfills all presented requirements, making reliable benchmarking and resource measurement easy. Our framework is able to work with a wide range of different tools, has proven its reliability and usefulness in the International Competition on Software Verification, and is used by several research groups worldwide to ensure reliable benchmarking. Finally, we present guidelines on how to present measurement results in a scientifically valid and comprehensible way.