Empirically revisiting and enhancing IR-based test-case prioritization
Empirically revisiting and enhancing IR-based test-case prioritization
复制标题
DOI：
10.1145/3395363.3397383
复制
发表时间：
2020-07
期刊：
影响因子：
--
通讯作者：
Qianyang Peng;A. Shi;Lingming Zhang
中科院分区：
文献类型：
--
作者：
Qianyang Peng;A. Shi;Lingming Zhang
测试用例优先级（TCP）旨在通过重新排序测试运行来更快地检测回归错误。虽然TCP已经研究了20多年，但几乎总是使用种子故障/突变体进行评估，而不是使用真实的测试故障。在这项工作中，我们研究了最近用于TCP的变化感知信息检索（IR）技术。先前的工作表明，它比传统的基于覆盖率的TCP技术表现更好，但它只在一个小规模的数据集上进行了评估，该数据集具有基于种子故障/突变体的成本未知指标。我们通过进行更大和更现实的评估来扩展先前的工作，并提出了大幅提高性能的增强措施。特别是，我们评估原始技术的大规模，真实世界的软件进化数据集与真实的故障使用成本感知和成本不知情的指标在各种配置下。此外，我们设计和评估混合技术相结合的IR功能，历史测试执行时间，测试失败的频率。我们的研究结果表明，变化感知IR技术优于最先进的覆盖为基础的技术，在这个现实世界的设置，我们的混合技术，甚至进一步提高了原来的IR技术。此外，我们表明，TCP测试有很大的影响，评估的变化感知TCP技术的基础上真实的测试失败。
Test-case prioritization (TCP) aims to detect regression bugs faster via reordering the tests run. While TCP has been studied for over 20 years, it was almost always evaluated using seeded faults/mutants as opposed to using real test failures. In this work, we study the recent change-aware information retrieval (IR) technique for TCP. Prior work has shown it performing better than traditional coverage-based TCP techniques, but it was only evaluated on a small-scale dataset with a cost-unaware metric based on seeded faults/mutants. We extend the prior work by conducting a much larger and more realistic evaluation as well as proposing enhancements that substantially improve the performance. In particular, we evaluate the original technique on a large-scale, real-world software-evolution dataset with real failures using both cost-aware and cost-unaware metrics under various configurations. Also, we design and evaluate hybrid techniques combining the IR features, historical test execution time, and test failure frequencies. Our results show that the change-aware IR technique outperforms stateof-the-art coverage-based techniques in this real-world setting, and our hybrid techniques improve even further upon the original IR technique. Moreover, we show that flaky tests have a substantial impact on evaluating the change-aware TCP techniques based on real test failures.