Assessing test quality
Assessing test quality
复制标题
评估测试质量
DOI：
--
复制
发表时间：
2011
期刊：
影响因子：
--
通讯作者：
David Schuler
中科院分区：
文献类型：
--
作者：
David Schuler
在开发测试时，人们有兴趣创建高质量的测试来彻底测试程序。这项工作展示了如何通过影响指标的突变测试以及检查的覆盖率来评估测试质量。尽管影响测试质量的因素有很多，但最重要的因素是其揭示缺陷的能力，因为软件测试通常是为了检测缺陷而进行的。为此，测试必须提供在导致感染的条件下执行有缺陷代码的输入。这种感染必须传播并导致失败，这可以通过测试检查来检测。过去，测试输入质量方面得到了广泛的研究，而检查的质量却受到较少的关注。评估测试套件检查质量的传统方法是突变测试。突变测试将人为缺陷（突变）植入程序中，并检查测试是否检测到它们。虽然这种技术可以有效地评估检查的质量，但它也有两个缺点。首先，它对计算资源提出了巨大的需求。其次，等效突变体（即在语义上与原始程序等效的突变体）会降低结果的质量。在这项工作中，我们解决了这两个问题。我们提出了 JAVALANCHE 框架，该框架应用了多种优化来实现对现实程序的自动化和高效的突变测试。此外，我们通过引入影响指标来检测非等效突变体来解决等效突变体的问题。影响指标将在原始程序上运行的测试套件的属性与在变异版本上运行的测试套件的属性进行比较，并且基于程序运行的抽象，例如动态不变量、覆盖语句和返回值。这些指标的目的是对程序运行有更严重影响的突变更有可能是不等价的。此外，我们引入了检查覆盖率，这是衡量测试套件检查质量的另一种方法。通过从测试套件的所有显式检查中计算动态向后切片，检查的覆盖率确定了代码中不仅被执行的部分，而且实际上对测试套件检查的结果有贡献的部分。
When developing tests, one is interested in creating tests of good quality that thoroughly test the program. This work shows how to assess test quality through mutation testing with impact metrics, and through checked coverage. Although there a different aspects that contribute to a test’s quality, the most important factor is its ability to reveal defects, because software testing is usually carried out with the aim to detect defects. For this purpose, a test has to provide inputs that execute the defective code under such conditions that it causes an infection. This infection has to propagate and result in a failure, which can be detected by a check of the test. In the past, the aspect of test input quality has been extensively studied while the quality of checks has received less attention. The traditional way of assessing the quality of a test suite’s checks is mutation testing. Mutation testing seeds artificial defects (mutations) into a program, and checks whether the tests detect them. While this technique effectively assesses the quality of checks, it also has two drawbacks. First, it places a huge demand on computing resources. Second, equivalent mutants, which are mutants that are semantically equivalent to the original program, dilute the quality of the results. In this work, we address both of these issues. We present the JAVALANCHE framework that applies several optimizations to enable automated and efficient mutation testing for real-life programs. Furthermore, we address the problem of equivalent mutants by introducing impact metrics to detect non-equivalent mutants. Impact metrics compare properties of tests suite runs on the original program with runs on mutated versions, and are based on abstractions over program runs such as dynamic invariants, covered statements, and return values. The intention of these metrics is that mutations that have a graver influence on the program run are more likely to be non-equivalent. Moreover, we introduce checked coverage, an alternative approach to measure the quality of a test suite’s checks. Checked coverage determines the parts of the code that were not only executed, but that actually contribute to the results checked by the test suite, by computing dynamic backward slices from all explicit checks of the test suite.
DOI：
10.1109/icstw.2011.57
发表时间：
2011-03
期刊：
2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops
影响因子：
--
作者：
Jaechang Nam;David Schuler;A. Zeller
通讯作者：
Jaechang Nam;David Schuler;A. Zeller
DOI：
10.1109/icst.2011.32
发表时间：
2011-03
期刊：
2011 Fourth IEEE International Conference on Software Testing, Verification and Validation
影响因子：
--
作者：
David Schuler;A. Zeller
通讯作者：
David Schuler;A. Zeller