The role and value of replication in empirical software engineering results
The role and value of replication in empirical software engineering results
复制标题
DOI：
10.1016/j.infsof.2018.01.006
复制
发表时间：
2018-07
期刊：
影响因子：
--
通讯作者：
M. Shepperd;N. Ajienka;S. Counsell
中科院分区：
文献类型：
--
作者：
M. Shepperd;N. Ajienka;S. Counsell
许多方面都提出了关于实证研究结果的可靠性的担忧，这包括软件工程。有人建议将复制作为增加信心的重要手段。目的：我们旨在更好地理解重复性研究的价值，重复性研究与原始研究之间的确认水平，确认在统计学意义上意味着什么，以及哪些因素改变了这种关系。方法我们进行了系统回顾，以确定在(i)软件项目工作量预测和（ii）结对编程领域的相关复制实验研究。如果提供了足够的细节，我们计算预测区间。我们的综述找到了28篇独特的文章，这些文章描述了35项原始研究的重复，这些研究解决了75个研究问题。其中10个是外部复制，15个是内部复制，3个是内部同篇复制。内部与外部（由独立研究人员进行）获得“验证性”结果的比值比为8.64。我们还发现不完整的报告阻碍了我们提取效应大小估计的能力。在我们能够计算复制预测区间的地方，这些区间大得惊人。我们表明，有大量证据表明，目前的方法经验复制是非常有问题的。人们一致认为重复研究很重要，但需要更好地报道原始研究和重复研究。考虑到许多原始研究的低功率和不完整的报告，复制的确认程度以及它为软件工程社区带来的额外知识的程度可能是不清楚的。我们建议将注意力从复制研究转向元分析。
ContextConcerns have been raised from many quarters regarding the reliability of empirical research findings and this includes software engineering. Replication has been proposed as an important means of increasing confidence.ObjectiveWe aim to better understand the value of replication studies, the level of confirmation between replication and original studies, what confirmation means in a statistical sense and what factors modify this relationship.MethodWe perform a systematic review to identify relevant replication experimental studies in the areas of (i) software project effort prediction and (ii) pair programming. Where sufficient details are provided we compute prediction intervals.ResultsOur review locates 28 unique articles that describe replications of 35 original studies that address 75 research questions. Of these 10 are external, 15 internal and 3 internal-same-article replications. The odds ratio of internal to external (conducted by independent researchers) replications of obtaining a ‘confirmatory’ result is 8.64. We also found incomplete reporting hampered our ability to extract estimates of effect sizes. Where we are able to compute replication prediction intervals these were surprisingly large.ConclusionWe show that there is substantial evidence to suggest that current approaches to empirical replications are highly problematic. There is a consensus that replications are important, but there is a need for better reporting of both original and replicated studies. Given the low power and incomplete reporting of many original studies, it can be unclear the extent to which a replication is confirmatory and to what extent it yields additional knowledge to the software engineering community. We recommend attention is switched from replication research to meta-analysis.