Do Pseudo Test Suites Lead to Inflated Correlation in Measuring Test Effectiveness?
Do Pseudo Test Suites Lead to Inflated Correlation in Measuring Test Effectiveness?
复制标题
DOI：
10.1109/icst.2019.00033
复制
发表时间：
2019-04
期刊：
影响因子：
--
通讯作者：
Jie M. Zhang;Lingming Zhang;Dan Hao;Meng Wang;Lu Zhang
中科院分区：
文献类型：
--
作者：
Jie M. Zhang;Lingming Zhang;Dan Hao;Meng Wang;Lu Zhang
在软件质量保证中，代码覆盖率是衡量测试有效性的最广泛采用的标准。覆盖标准的性能（表示测试套件的有效性）在以前的工作中得到了广泛的研究。大多数研究使用随机构建的伪测试套件来方便相关分析的数据收集，但尚未有工作系统地研究伪测试套件是否会导致夸大的相关结果。本文通过对123个真实Java项目的研究来关注潜在的广泛威胁。遵循研究覆盖率标准的典型实验过程，我们使用伪测试套件和原始测试套件研究语句/断言覆盖率与突变分数之间的相关性。除了直接相关分析外，我们还控制断言的数量和测试套件的大小来进行部分相关分析。结果表明：1)伪测试套件与原始测试套件的相关性（覆盖标准与突变评分之间的相关性）显著高于原始测试套件（Kendall值高0.21 ~ 0.39）；2)与先前报道相反，语句覆盖率与突变分数的相关性比断言覆盖率更强。
Code coverage is the most widely adopted criteria for measuring test effectiveness in software quality assurance. The performance of coverage criteria (in indicating test suites' effectiveness) has been widely studied in prior work. Most of the studies use randomly constructed pseudo test suites to facilitate data collection for correlation analysis, yet no previous work has systematically studied whether pseudo test suites would lead to inflated correlation results. This paper focuses on the potentially wide-spread threat with a study over 123 real-world Java projects. Following the typical experimental process of studying coverage criteria, we investigate the correlation between statement/assertion coverage and mutation score using both pseudo and original test suites. Except for direct correlation analysis, we control the number of assertions and the test suite size to conduct partial correlation analysis. The results reveal that 1) the correlation (between coverage criteria and mutation score) derived from pseudo test suites is much higher than from original test suites (from 0.21 to 0.39 higher in Kendall value); 2) contrary to previously reported, statement coverage has a stronger correlation with mutation score than assertion coverage.