Does automated white-box test generation really help software testers?
Does automated white-box test generation really help software testers?
复制标题
DOI：
10.1145/2483760.2483774
复制
发表时间：
2013-07
期刊：
影响因子：
--
通讯作者：
G. Fraser;Matthew Staats;Phil McMinn;Andrea Arcuri;F. Padberg
中科院分区：
文献类型：
--
作者：
G. Fraser;Matthew Staats;Phil McMinn;Andrea Arcuri;F. Padberg
自动化测试生成技术可以有效地生成系统涵盖程序结构方面的测试数据。在没有规范的情况下，一个共同的假设是这些测试可以减轻开发人员的大多数工作，因为测试行为减少到检查测试结果。尽管这种假设持续了数十年，但迄今为止没有确切的证据证明这一点。但是，该方法只有在行业中的吸收有限的事实表明了相反，并质疑其实际实用性。为了调查此问题，我们进行了一个受控的实验，比较了手动编写测试和借助自动单位测试生成工具EvoSuite在手动编写测试和编写测试之间分裂的49名受试者。我们发现，一方面，工具支持可以明确改善常用质量指标，例如代码覆盖率（增加300％）。但是，另一方面，开发人员实际发现的错误数量没有可测量的改善。我们的结果不仅对研究界如何评估测试生成工具的评估以及在自动化测试生成工具被从业者广泛采用之前的改进和未来工作的疑问。
Automated test generation techniques can efficiently produce test data that systematically cover structural aspects of a program. In the absence of a specification, a common assumption is that these tests relieve a developer of most of the work, as the act of testing is reduced to checking the results of the tests. Although this assumption has persisted for decades, there has been no conclusive evidence to date confirming it. However, the fact that the approach has only seen a limited uptake in industry suggests the contrary, and calls into question its practical usefulness. To investigate this issue, we performed a controlled experiment comparing a total of 49 subjects split between writing tests manually and writing tests with the aid of an automated unit test generation tool, EvoSuite. We found that, on one hand, tool support leads to clear improvements in commonly applied quality metrics such as code coverage (up to 300% increase). However, on the other hand, there was no measurable improvement in the number of bugs actually found by developers. Our results not only cast some doubt on how the research community evaluates test generation tools, but also point to improvements and future work necessary before automated test generation tools will be widely adopted by practitioners.