Repeatability of systematic literature reviews
Repeatability of systematic literature reviews
复制标题
系统文献综述的可重复性
DOI：
10.1049/ic.2011.0006
复制
发表时间：
2011
期刊：
影响因子：
--
通讯作者：
Kitchenham B
中科院分区：
文献类型：
--
作者：
Kitchenham B
背景系统文献回顾(SLR)的预期好处之一是，它们可以以可审计的方式进行，以产生可重复的结果。目的本研究的目的是确定在什么条件下SLR在所选的主要研究中用于软件工程时可能是稳定的。我们在这份报告中调查的条件是新手研究人员为了共同的目标进行搜索。方法我们进行了参与者-观察者多案例研究，以考察系统文献综述的可重复性。这项研究中的“案例”是两种单元测试方法的SLR的早期阶段，涉及到相关文献的识别。单反由两名新手研究人员独立进行。SLR仅限于ACM和IEEE数字图书馆1986-2005年，因此他们的结果可以与发表的单元测试论文的专家文献综述进行比较。结果这两个SLR选择了非常不同的论文，32篇论文中只有6篇是相同的，而且两者都与发表的单元测试论文的二次研究结果有很大差异，21篇论文中只有3篇。在新手研究人员发现的另外29篇论文中，只有10篇被认为是相关的。增加的10篇相关论文将对已发表的研究结果产生影响，因为它们在框架中增加了三个新的类别，并将论文添加到三个否则为空的单元格中。结论对于新手研究人员来说，具有大致相同的研究问题并不一定保证初步研究的可重复性。系统审查必须小心地完整地报告他们的搜索过程，否则它们将无法重复。论文缺失可能会对二次研究结果的稳定性产生重大影响。
BackgroundOne of the anticipated benefits of systematic literature reviews (SLRs) is that they can be conducted in an auditable way to produce repeatable results.AimThis study aims to identify under what conditions SLRs are likely to be stable, with respect to the primary studies selected, when used in software engineering. The conditions we investigate in this report are when novice researchers undertake searches with a common goal.MethodWe undertook a participant-observer multi-case study to investigate the repeatability of systematic literature reviews. The "cases" in this study were the early stages, involving identification of relevant literature, of two SLRs of unit testing methods. The SLRs were performed independently by two novice researchers. The SLRs were restricted to the ACM and IEEE digital libraries for the years 1986-2005 so their results could be compared with a published expert literature review of unit testing papers.ResultsThe two SLRs selected very different papers with only six papers out of 32 in common, and both differed substantially from a published secondary study of unit testing papers finding only three of 21 papers. Of the 29 additional papers found by the novice researchers, only 10 were considered relevant. The 10 additional relevant papers would have had an impact on the results of the published study by adding three new categories to the framework and adding papers to three, otherwise empty, cells.ConclusionsIn the case of novice researchers, having broadly the same research question will not necessarily guarantee repeatability with respect to primary studies. Systematic reviews must be careful to report their search process fully or they will not be repeatable. Missing papers can have a significant impact on the stability of the results of a secondary study.
登录
查看更多内容
影响因子：
4.1
作者：
Natalia Juristo Juzgado;A. Moreno;S. Vegas
通讯作者：
S. Vegas
数据覆盖测试
DOI：
--
发表时间：
2002
期刊：
Ninth Asia-Pacific Software Engineering Conference, 2002.
影响因子：
--
作者：
P. Netisopakul;L. White;J. Morris
通讯作者：
J. Morris
DOI：
10.1145/1806799.1806887
发表时间：
2010
期刊：
2010 ACM/IEEE 32nd International Conference on Software Engineering
影响因子：
--
作者：
B. Kitchenham;P. Brereton;D. Budgen
通讯作者：
D. Budgen
实证软件工程中的系统评审有多可靠？
影响因子：
7.4
作者：
Stephen G. MacDonell;M. Shepperd;B. Kitchenham;E. Mendes
通讯作者：
E. Mendes
突变和数据流测试的故障检测有效性
影响因子：
1.9
作者：
W.;Eric;’. Wong;Aditya;P.;Mathur
通讯作者：
Mathur