Reproducibility and credibility in empirical software engineering: A case study based on a systematic literature review of the use of the SZZ algorithm
Reproducibility and credibility in empirical software engineering: A case study based on a systematic literature review of the use of the SZZ algorithm
复制标题
DOI：
10.1016/j.infsof.2018.03.009
复制
发表时间：
2018-07
期刊：
影响因子：
--
通讯作者：
Gema Rodríguez-Pérez;G. Robles;Jesus M. Gonzalez-Barahona
中科院分区：
文献类型：
--
作者：
Gema Rodríguez-Pérez;G. Robles;Jesus M. Gonzalez-Barahona
背景经验软件工程研究的可重复性是提高其可信度的重要组成部分，因为它为研究社区提供了验证、评估和改进其研究成果的机会。目的我们通过一个案例研究来研究经验软件工程研究的可重复性和可信度，通过调查它们是如何在应用SZZ的研究中得到解决的，SZZ是Śliwerski，zzmermann和zeller广泛使用的一种检测错误来源的算法。总共分析了187篇论文的重复性、报告的局限性和使用改进版本的算法。结果我们发现ESE存在着很大的改进空间，因为重复性并不常见；破坏结果可信度的因素很常见。我们提供了一些经验教训和指导方针，供研究人员和评审者解决这个问题。结论ESE社区应该更多地考虑重复性和其他相关方面，以确保高质量的科学过程，以增加研究结果的可信度。
ContextReproducibility of Empirical Software Engineering (ESE) studies is an essential part for improving their credibility, as it offers the opportunity to the research community to verify, evaluate and improve their research outcomes.ObjectiveWe aim to study reproducibility and credibility in ESE with a case study, by investigating how they have been addressed in studies where SZZ, a widely-used algorithm by Śliwerski, Zimmermann and Zeller to detect the origin of a bug, has been applied.MethodologyWe have performed a systematic literature review to evaluate publications that use SZZ. In total, 187 papers have been analyzed for reproducibility, reporting of limitations and use of improved versions of the algorithm.ResultsWe have found a situation with a lot of room for improvement in ESE as reproducibility is not commonly found; factors that undermine the credibility of results are common. We offer some lessons learned and guidelines for researchers and reviewers to address this problem.ConclusionReproducibility and other related aspects that ensure a high quality scientific process should be taken more into consideration by the ESE community in order to increase the credibility of the research results.