On the use of mutation analysis for evaluating student test suite quality
On the use of mutation analysis for evaluating student test suite quality
复制标题
关于使用突变分析来评估学生测试套件的质量
DOI：
10.1145/3533767.3534217
复制
发表时间：
2022
期刊：
影响因子：
--
通讯作者：
Bell, Jonathan
中科院分区：
文献类型：
--
作者：
Perretta, James;DeOrio, Andrew;Guha, Arjun;Bell, Jonathan
计算机科学课程中的一个常见做法是根据一组手动播种的错误（由教师手写）或所有其他学生编写的实现（“所有对”评分）来评估学生编写的测试套件。然而，手动播种故障是一个耗时且可能容易出错的过程，并且全对方法需要大量的手动和计算工作才能公平准确地应用。突变分析，自动种子的实现中的潜在故障，是一个可能的替代这些测试套件评估方法。虽然在文献中有证据表明突变体是大型开源软件项目中真实的错误的有效替代品，但尚不清楚突变体是否代表学生所犯的错误。如果突变体是学生编写的代码中发现的错误的有效替代品，并且如果突变体检测与手动播种的错误检测和错误的学生实现检测相关，那么教师可以使用开源突变分析工具生成的突变体来评估学生测试套件。使用2，711个学生作业提交的数据集，我们凭经验评估突变分数是否是手动播种错误检测率和错误学生实现检测率的良好代理。我们的研究结果表明，突变分数和手动播种的故障检测率之间有很强的相关性，突变分数和错误的学生实现检测之间有中等强度的相关性。我们确定了少数故障的学生实现，耦合到一个突变体，将需要新的或更强的突变算子或应用突变算子的实现具有不同的结构比的突变器编写的实现。我们还发现，这种相关性是有限的事实，故障分布不均匀，在整个学生代码，所有对分级的一个已知的缺点。我们的研究结果表明，由开源突变分析工具产生的突变体具有相同或更高的质量比手动播种的故障和一个相当好的替身真实的故障在学生实现。我们的发现对软件测试研究人员、教育工作者和工具开发者都有意义。
A common practice in computer science courses is to evaluate student-written test suites against either a set of manually-seeded faults (handwritten by an instructor) or against all other student-written implementations (“all-pairs” grading). However, manually seeding faults is a time consuming and potentially error-prone process, and the all-pairs approach requires significant manual and computational effort to apply fairly and accurately. Mutation analysis, which automatically seeds potential faults in an implementation, is a possible alternative to these test suite evaluation approaches. Although there is evidence in the literature that mutants are a valid substitute for real faults in large open-source software projects, it is unclear whether mutants are representative of the kinds of faults that students make. If mutants are a valid substitute for faults found in student-written code, and if mutant detection is correlated with manually-seeded fault detection and faulty student implementation detection, then instructors can instead evaluate student test suites using mutants generated by open-source mutation analysis tools.Using a dataset of 2,711 student assignment submissions, we empirically evaluate whether mutation score is a good proxy for manually-seeded fault detection rate and faulty student implementation detection rate. Our results show a strong correlation between mutation score and manually-seeded fault detection rate and a moderately strong correlation between mutation score and faulty student implementation detection. We identify a handful of faults in student implementations that, to be coupled to a mutant, would require new or stronger mutation operators or applying mutation operators to an implementation with a different structure than the instructor-written implementation. We also find that this correlation is limited by the fact that faults are not distributed evenly throughout student code, a known drawback of all-pairs grading. Our results suggest that mutants produced by open-source mutation analysis tools are of equal or higher quality than manually-seeded faults and a reasonably good stand-in for real faults in student implementations. Our findings have implications for software testing researchers, educators, and tool builders alike.
登录
查看更多内容
DOI：
10.1145/1029994.1029995
发表时间：
2003-09
期刊：
ACM J. Educ. Resour. Comput.
影响因子：
--
作者：
S. Edwards
通讯作者：
S. Edwards
用于评估学生编写的软件测试质量的实用突变分析
DOI：
10.1145/2493394.2493402
发表时间：
2013
期刊：
Proceedings of the ninth annual international ACM conference on International computing education research
影响因子：
--
作者：
Z. Shams;S. Edwards
通讯作者：
S. Edwards
PIT：实用的 Java 突变测试工具（演示）
DOI：
--
发表时间：
2016
期刊：
International Symposium on Software Testing and Analysis
影响因子：
--
作者：
Henry Coles;Thomas Laurent;Christopher Henard;Mike Papadakis;Anthony Ventresque
通讯作者：
Anthony Ventresque
DOI：
10.1016/s0020-7373(83)80061-3
发表时间：
1983
期刊：
Int. J. Man Mach. Stud.
影响因子：
--
作者：
M. Weiser;Joan Shertz
通讯作者：
Joan Shertz
比较评估学生书面测试的测试质量措施
DOI：
10.1145/2591062.2591164
发表时间：
2014
期刊：
Companion Proceedings of the 36th International Conference on Software Engineering
影响因子：
--
作者：
S. Edwards;Z. Shams
通讯作者：
Z. Shams