Empirically revisiting the test independence assumption
Empirically revisiting the test independence assumption
复制标题
根据经验重新审视测试独立性假设
DOI：
--
复制
发表时间：
2014
期刊：
影响因子：
--
通讯作者：
D. Notkin
中科院分区：
文献类型：
--
作者：
Sai Zhang;D. Jalali;Jochen Wuttke;Kivanç Muslu;Wing Lam;Michael D. Ernst;D. Notkin
在一个测试套件中，所有的测试用例都应该是独立的：任何测试都不应该影响任何其他测试的结果，并且以任何顺序运行测试都应该产生相同的测试结果。诸如测试优先级之类的技术通常假设测试套件中的测试是独立的。测试依赖是一个很少被研究的现象。本文介绍了五个结果有关的测试依赖。
首先，我们描述了在实践中出现的测试依赖。我们研究了来自5个问题跟踪系统的96个真实依赖测试。我们的研究表明，测试依赖可能很难为程序员识别。它还表明，测试依赖可能会导致不平凡的后果，如掩盖程序错误，并导致虚假的错误报告。
其次，我们正式定义测试依赖的测试套件作为有序序列的测试沿着与明确的环境中，这些测试的执行。我们制定的问题，检测相关的测试，并证明了一个有用的特殊情况下是NP完全的。
第三，通过对真实世界中相关测试的研究，我们提出并比较了四种检测相关测试的算法。
第四，我们将我们的依赖测试检测算法应用于4个真实世界的程序，并在每个人类编写和自动生成的测试套件中发现依赖测试。
第五，我们经验性地评估了五种测试优先化技术对依赖测试的影响。依赖测试影响所有五种技术的输出;也就是说，即使原始套件没有失败，重新排序的套件也会失败。
In a test suite, all the test cases should be independent: no test should affect any other test’s result, and running the tests in any order should produce the same test results. Techniques such as test prioritization generally assume that the tests in a suite are independent. Test dependence is a little-studied phenomenon. This paper presents five results related to test dependence.
First, we characterize the test dependence that arises in practice. We studied 96 real-world dependent tests from 5 issue tracking systems. Our study shows that test dependence can be hard for programmers to identify. It also shows that test dependence can cause non-trivial consequences, such as masking program faults and leading to spurious bug reports.
Second, we formally define test dependence in terms of test suites as ordered sequences of tests along with explicit environments in which these tests are executed. We formulate the problem of detecting dependent tests and prove that a useful special case is NP-complete.
Third, guided by the study of real-world dependent tests, we propose and compare four algorithms to detect dependent tests in a test suite.
Fourth, we applied our dependent test detection algorithms to 4 real-world programs and found dependent tests in each human-written and automatically-generated test suite.
Fifth, we empirically assessed the impact of dependent tests on five test prioritization techniques. Dependent tests affect the output of all five techniques; that is, the reordered suite fails even though the original suite did not.
DOI：
10.1109/icse.2007.18
发表时间：
2007-05
期刊：
29th International Conference on Software Engineering (ICSE'07)
影响因子：
--
作者：
Zhimin Wang;Sebastian G. Elbaum;David S. Rosenblum
通讯作者：
Zhimin Wang;Sebastian G. Elbaum;David S. Rosenblum