On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques
On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques
复制标题
DOI：
10.1109/tse.2006.92
复制
发表时间：
2006-09
影响因子：
7.4
通讯作者：
Hyunsook Do;G. Rothermel
中科院分区：
文献类型：
--
作者：
Hyunsook Do;G. Rothermel
回归测试是软件生命周期中的一项重要活动，但也可能非常昂贵。为了降低回归测试的成本，软件测试人员可能会对他们的测试用例进行优先排序，以便在回归测试过程中更早地运行那些更重要的用例。测试用例优先技术的一个潜在目标是提高测试套件的故障检测率(在测试用例的运行中，该测试套件检测故障的速度有多快)。以前的工作已经表明，优先排序可以提高测试集的故障检测率，但优先排序技术的评估主要限于手动播种的错误，这主要是因为这样的错误比自动生成(突变)的错误更现实。然而，最近的一项经验研究表明，突变故障可以代表真实的故障，并且使用人工播种的故障可能会对侧重于故障检测的经验结果的有效性产生问题。因此，我们设计并执行了两个对照实验，评估优先排序技术的能力，以提高测试用例优先排序技术的错误检测率，相对于突变错误进行测量。我们的结果表明，相对于所考虑的故障，优先级排序可能是有效的，并且它们揭示了这种有效性可以随故障和测试套件的特征而变化的方式。更重要的是，将我们的结果与使用手工播种的错误收集的结果进行比较，可以为执行测试用例优先排序技术和一般测试技术的研究人员提供几点启示
Regression testing is an important activity in the software life cycle, but it can also be very expensive. To reduce the cost of regression testing, software testers may prioritize their test cases so that those which are more important, by some measure, are run earlier in the regression testing process. One potential goal of test case prioritization techniques is to increase a test suite's rate of fault detection (how quickly, in a run of its test cases, that test suite can detect faults). Previous work has shown that prioritization can improve a test suite's rate of fault detection, but the assessment of prioritization techniques has been limited primarily to hand-seeded faults, largely due to the belief that such faults are more realistic than automatically generated (mutation) faults. A recent empirical study, however, suggests that mutation faults can be representative of real faults and that the use of hand-seeded faults can be problematic for the validity of empirical results focusing on fault detection. We have therefore designed and performed two controlled experiments assessing the ability of prioritization techniques to improve the rate of fault detection of test case prioritization techniques, measured relative to mutation faults. Our results show that prioritization can be effective relative to the faults considered, and they expose ways in which that effectiveness can vary with characteristics of faults and test suites. More importantly, a comparison of our results with those collected using hand-seeded faults reveals several implications for researchers performing empirical studies of test case prioritization techniques in particular and testing techniques in general