On the Danger of Coverage Directed Test Case Generation
On the Danger of Coverage Directed Test Case Generation
复制标题
DOI：
10.1007/978-3-642-28872-2_28
复制
发表时间：
2012-03
期刊：
影响因子：
--
通讯作者：
Matthew Staats;Gregory Gay;M. Whalen;M. Heimdahl
中科院分区：
文献类型：
--
作者：
Matthew Staats;Gregory Gay;M. Whalen;M. Heimdahl
在航空电子领域，在确定测试套件充分性时，法律上要求使用结构覆盖标准。随着自动化测试生成工具的成功，使用这些标准作为测试生成的基础是很有诱惑力的。为了更好地确定这种方法的有效性，我们使用基于反例的测试生成和随机生成方法生成并评估了满足两个覆盖标准的测试用例，并与同等大小的纯随机测试用例进行了对比。我们的结果得出了两个关键结论。首先，仅满足覆盖率标准本身并不能很好地说明测试套件的有效性。其次，使用结构性覆盖作为测试生成的补充--而不是目标--可能会产生积极的影响。这些观察指出了关键系统中测试自动化增加所固有的危险，以及需要更多地研究覆盖标准、生成方法和系统结构如何共同影响测试有效性。
In the avionics domain, the use of structural coverage criteria is legally required in determining test suite adequacy. With the success of automated test generation tools, it is tempting to use these criteria as the basis for test generation. To more firmly establish the effectiveness of such approaches, we have generated and evaluated test suites to satisfy two coverage criteria using counterexample-based test generation and a random generation approach, contrasted against purely random test suites of equal size.Our results yield two key conclusions. First, coverage criteria satisfaction alone is a poor indication of test suite effectiveness. Second, the use of structural coverage as a supplement—not a target—for test generation can have a positive impact. These observations points to the dangers inherent in the increase in test automation in critical systems and the need for more research in how coverage criteria, generation approach, and system structure jointly influence test effectiveness.