An empirical evaluation and comparison of manual and automated test selection
An empirical evaluation and comparison of manual and automated test selection
复制标题
手动和自动测试选择的实证评估和比较
DOI：
--
复制
发表时间：
2014
期刊：
影响因子：
--
通讯作者：
D. Marinov
中科院分区：
文献类型：
--
作者：
Miloš Gligorić;Stas Negara;Owolabi Legunsen;D. Marinov
回归测试选择仅通过重新运行可能会受到最新代码更改影响的测试来加快回归测试。在过去的三十年中，在自动测试选择方面的研究取得了很多进展，但尚未转化为被广泛采用的实用工具。因此，开发人员要么在每次更改后重新运行所有测试，要么执行手动测试选择。重新运行所有测试都是昂贵的，而手动测试的选择则乏味且容易出错。尽管经过如此巨大的权衡，但没有研究评估了开发人员如何进行手动测试的选择，并将其与自动测试选择进行了比较。本文报告了我们在实践中对手动测试选择的研究以及对手动和自动测试选择的比较。我们是第一个进行研究的研究，该研究（1）分析手动测试选择的数据，该数据在三个月的研究中从14个开发人员实时收集，（2）将手动测试选择与自动化的最新研究进行了比较450次测试课程的测试选择工具。我们研究中的几乎所有开发人员均进行了手动测试选择，他们大多以临时的方式进行了选择。比较手动和自动测试选择，我们发现了在研究的450个测试课程中的每一种中选择不同测试的两种方法。手动选择比自动选择的时间多73％（可能浪费时间），并且选择了27％的时间（可能缺少错误的错误）。这些结果表明，需要更好地集成到开发人员的编程环境中，需要更好的自动测试选择技术。
Regression test selection speeds up regression testing by re-running only the tests that can be affected by the most recent code changes. Much progress has been made on research in automated test selection over the last three decades, but it has not translated into practical tools that are widely adopted. Therefore, developers either re-run all tests after each change or perform manual test selection. Re-running all tests is expensive, while manual test selection is tedious and error-prone. Despite such a big trade-off, no study assessed how developers perform manual test selection and compared it to automated test selection. This paper reports on our study of manual test selection in practice and our comparison of manual and automated test selection. We are the first to conduct a study that (1) analyzes data from manual test selection, collected in real time from 14 developers during a three-month study and (2) compares manual test selection with an automated state-of-the-research test-selection tool for 450 test sessions. Almost all developers in our study performed manual test selection, and they did so in mostly ad-hoc ways. Comparing manual and automated test selection, we found the two approaches to select different tests in each and every one of the 450 test sessions investigated. Manual selection chose more tests than automated selection 73% of the time (potentially wasting time) and chose fewer tests 27% of the time (potentially missing bugs). These results show the need for better automated test-selection techniques that integrate well with developers' programming environments.