Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges (T)
Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges (T)
复制标题
DOI：
10.1109/ase.2015.86
复制
发表时间：
2015-11
期刊：
影响因子：
--
通讯作者：
S. Shamshiri;René Just;J. Rojas;G. Fraser;Phil McMinn;Andrea Arcuri
中科院分区：
文献类型：
--
作者：
S. Shamshiri;René Just;J. Rojas;G. Fraser;Phil McMinn;Andrea Arcuri
可以使用工具自动生成它们，而不是乏味地编写单元测试 - 有时甚至导致代码覆盖范围更高，而不是手动测试。但是，这些测试实际发现故障的测试有多好？要回答这个问题，我们将Java（Randoop，EvoSuite和Agitar）的三个最先进的单元测试生成工具应用于缺陷4J数据集中的357个真正的故障，并研究了生成的测试套件在检测这些测试套件方面的表现如何故障。尽管自动生成的测试套件总体上检测到了55.7％的故障，但只有19.9％的个人测试套件检测到了故障。通过研究单个工具的有效性和问题以及它们产生的测试，我们提供了见解，以支持实现较高故障检测率的自动化单元测试发电机的开发。这些见解包括1）提高所获得的代码覆盖范围，以便在第一案中执行错误的陈述，2）将错误的程序状态的传播提高到可观察的输出，再加上更敏感的断言，以及3）改进模拟执行环境以检测取决于外部因素（例如日期和时间）的故障。
Rather than tediously writing unit tests manually, tools can be used to generate them automatically - sometimes even resulting in higher code coverage than manual testing. But how good are these tests at actually finding faults? To answer this question, we applied three state-of-the-art unit test generation tools for Java (Randoop, EvoSuite, and Agitar) to the 357 real faults in the Defects4J dataset and investigated how well the generated test suites perform at detecting these faults. Although the automatically generated test suites detected 55.7% of the faults overall, only 19.9% of all the individual test suites detected a fault. By studying the effectiveness and problems of the individual tools and the tests they generate, we derive insights to support the development of automated unit test generators that achieve a higher fault detection rate. These insights include 1) improving the obtained code coverage so that faulty statements are executed in the first instance, 2) improving the propagation of faulty program states to an observable output, coupled with the generation of more sensitive assertions, and 3) improving the simulation of the execution environment to detect faults that are dependent on external factors such as date and time.