An Industrial Application of Mutation Testing: Lessons, Challenges, and Research Directions
An Industrial Application of Mutation Testing: Lessons, Challenges, and Research Directions
复制标题
突变测试的工业应用：经验教训、挑战和研究方向
DOI：
--
复制
发表时间：
2018
期刊：
影响因子：
--
通讯作者：
René Just
中科院分区：
文献类型：
--
作者：
Goran Petrović;M. Ivankovic;Bob Kurtz;P. Ammann;René Just
突变分析通过测量其检测到系统播种的突变体的能力来评估测试或调试技术。由于其产生的突变体数量，突变分析本质上是昂贵的，并且由于许多产生的突变体无效的事实。它们是冗余，等效的，或者只是无趣和浪费计算资源。大量研究的重点是提高突变分析的可扩展性，并提出了许多优化，例如选择有效的突变体或有效地针对大量突变体进行大量测试。但是，在工业规模的软件开发过程的背景下，相对较少的研究集中在突变测试的成本和收益上，其中突变体被作为向开发人员进行测试目标。本文借鉴了突变测试的工业应用，涉及30,000多个开发人员和190万个更改集，并以4种编程语言编写。它表明，使用生产性突变体的突变测试并未在软件开发过程中添加重要的开销，并报告了开发人员感知的突变测试益处。本文还量化了非生产性突变体的成本，结果表明，达到突变充足性既不是实际也不可取的。最后，本文描述了从这些研究中学到的经验教训，强调了当前有效，有效地应用突变测试在工业规模的软件开发过程中的挑战，并概述了研究方向。
Mutation analysis evaluates a testing or debugging technique by measuring how well it detects mutants, which are systematically seeded, artificial faults. Mutation analysis is inherently expensive due to the large number of mutants it generates and due to the fact that many of these generated mutants are not effective; they are redundant, equivalent, or simply uninteresting and waste computational resources. A large body of research has focused on improving the scalability of mutation analysis and proposed numerous optimizations to, e.g., select effective mutants or efficiently execute a large number of tests against a large number of mutants. However, comparatively little research has focused on the costs and benefits of mutation testing, in which mutants are presented as testing goals to a developer, in the context of an industrial-scale software development process. This paper draws on an industrial application of mutation testing, involving 30,000+ developers and 1.9 million change sets, written in 4 programming languages. It shows that mutation testing with productive mutants does not add a significant overhead to the software development process and reports on mutation testing benefits perceived by developers. This paper also quantifies the costs of unproductive mutants, and the results suggest that achieving mutation adequacy is neither practical nor desirable. Finally, this paper describes lessons learned from these studies, highlights the current challenges of efficiently and effectively applying mutation testing in an industrial-scale software development process, and outlines research directions.
DOI：
10.1109/icse.2017.62
发表时间：
2017-05
期刊：
2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE)
影响因子：
--
作者：
Spencer Pearson;José Campos;René Just;G. Fraser;Rui Abreu;Michael D. Ernst;D. Pang;Benjamin Keller-Benjamin-K
通讯作者：
Spencer Pearson;José Campos;René Just;G. Fraser;Rui Abreu;Michael D. Ernst;D. Pang;Benjamin Keller-Benjamin-K