A large-scale study of test coverage evolution
A large-scale study of test coverage evolution
复制标题
测试覆盖率演变的大规模研究
DOI：
10.1145/3238147.3238183
复制
发表时间：
2018
期刊：
影响因子：
--
通讯作者：
Marinov, Darko
中科院分区：
文献类型：
--
作者：
Hilton, Michael;Bell, Jonathan;Marinov, Darko
语句覆盖率通常被用作测试套件质量的度量。覆盖率通常被用作代码审查过程的一部分：如果一个补丁减少了总体覆盖率，或者它本身没有被覆盖，那么这个补丁就会被更仔细地审查。关于覆盖率如何随代码演化而变化的传统研究检查了整个程序的总体覆盖率，而最近的工作则直接检查了补丁（更改语句）的覆盖率。我们提出了一个比以往研究大得多的评估，并且考虑了一种新的、重要的变化——未改变陈述的覆盖率变化。我们对47个用流行语言（包括Java、Python和Scala）编写的项目的7816个构建进行了大规模的代码覆盖演化评估。我们发现，在大型的、成熟的项目中，简单地度量语句覆盖率的变化并不能捕捉到代码演化的细微差别。除了将语句覆盖率视为一个简单的比率之外，我们还研究了在项目修订之间所涵盖的语句集是如何演变的。我们提出并研究了新的方法来评估补丁对项目测试套件质量的影响，这些方法既将补丁的覆盖与非补丁的覆盖分开，又将覆盖中的变化与覆盖的语句集中的变化分开。
Statement coverage is commonly used as a measure of test suite quality. Coverage is often used as a part of a code review process: if a patch decreases overall coverage, or is itself not covered, then the patch is scrutinized more closely. Traditional studies of how coverage changes with code evolution have examined the overall coverage of the entire program, and more recent work directly examines the coverage of patches (changed statements). We present an evaluation much larger than prior studies and moreover consider a new, important kind of change --- coverage changes of unchanged statements. We present a large-scale evaluation of code coverage evolution over 7,816 builds of 47 projects written in popular languages including Java, Python, and Scala. We find that in large, mature projects, simply measuring the change to statement coverage does not capture the nuances of code evolution. Going beyond considering statement coverage as a simple ratio, we examine how the set of statements covered evolves between project revisions. We present and study new ways to assess the impact of a patch on a project's test suite quality that both separates coverage of the patch from coverage of the non-patch, and separates changes in coverage from changes in the set of statements covered.