Does Automated Unit Test Generation Really Help Software Testers? A Controlled Empirical Study
Does Automated Unit Test Generation Really Help Software Testers? A Controlled Empirical Study
复制标题
DOI：
10.1145/2699688
复制
发表时间：
2015-09
期刊：
影响因子：
--
通讯作者：
G. Fraser;Matthew Staats;Phil McMinn;Andrea Arcuri;F. Padberg
中科院分区：
文献类型：
--
作者：
G. Fraser;Matthew Staats;Phil McMinn;Andrea Arcuri;F. Padberg
自动测试生成的工作已经生产了几种能够生成测试数据的工具，这些工具可以在程序上获得高结构覆盖。在没有规范的情况下，期望开发人员手动构建或验证每个测试输入的测试ORACLE。然而，假定这些生成的测试可以简化开发人员测试的任务，因为测试减少到检查测试结果。尽管这种假设持续了数十年，但迄今为止没有确切的证据证明这一点。但是，行业中的有限采用表明该假设可能不正确，并质疑测试生成工具的实际价值。为了调查此问题，我们进行了两个受控的实验，比较了手动编写测试和借助自动单位测试生成工具EvoSuite在手动编写测试和编写测试之间分裂的97名受试者。我们发现，一方面，工具支持可以明确改善常用质量指标，例如代码覆盖率（增加300％）。但是，另一方面，开发人员实际发现的错误数量没有可测量的改善。我们的结果不仅对研究界如何评估测试生成工具的评估以及在自动化测试生成工具被从业者广泛采用之前的改进和未来工作的疑问。
Work on automated test generation has produced several tools capable of generating test data which achieves high structural coverage over a program. In the absence of a specification, developers are expected to manually construct or verify the test oracle for each test input. Nevertheless, it is assumed that these generated tests ease the task of testing for the developer, as testing is reduced to checking the results of tests. While this assumption has persisted for decades, there has been no conclusive evidence to date confirming it. However, the limited adoption in industry indicates this assumption may not be correct, and calls into question the practical value of test generation tools. To investigate this issue, we performed two controlled experiments comparing a total of 97 subjects split between writing tests manually and writing tests with the aid of an automated unit test generation tool, EvoSuite. We found that, on one hand, tool support leads to clear improvements in commonly applied quality metrics such as code coverage (up to 300% increase). However, on the other hand, there was no measurable improvement in the number of bugs actually found by developers. Our results not only cast some doubt on how the research community evaluates test generation tools, but also point to improvements and future work necessary before automated test generation tools will be widely adopted by practitioners.