Reproducible Searches in Systematic Reviews: An Evaluation and Guidelines
Reproducible Searches in Systematic Reviews: An Evaluation and Guidelines
复制标题
DOI：
10.1109/access.2023.3299211
复制
发表时间：
2023
期刊：
影响因子：
3.9
通讯作者：
Zhengyang Li;A. Rainer
中科院分区：
文献类型：
--
作者：
Zhengyang Li;A. Rainer
[背景：]系统性综述被认为是对先前研究进行高质量综述的一种更可靠的方式。但有一系列威胁可能会破坏此类评论的可靠性和质量。一个威胁是自动搜索的可重复性。[目标：]评估二级研究中可重复性检索的实践现状，并考虑提高检索可重复性的方法。[方法：]我们重新运行了621项二级研究的搜索，并分析了这些（尝试）重新运行的结果。我们使用的结果，我们的经验，重新运行的搜索，提出方法来提高自动搜索的可重复性。[结果：]在621项研究中，超过50%的文字搜索字符串（忽略其他设置）不可重用;大约87%的搜索（例如，有设置）不能重复;大约94%的搜索（包括搜索的所有元素）是不可复制的。我们提出了自动搜索的指导方针，特别注意制定搜索字符串。[结论：]虽然自动搜索的某些方面超出了研究人员的直接控制（例如，搜索引擎的特征、约束和性能的变化），通过更仔细地制定和执行搜索字符串本身以及搜索设置，可以有效地管理许多方面。虽然我们的评估结果令人失望，但研究人员可以采取许多简单，具体的步骤来提高他们搜索的可重复性。
[Context:] The Systematic Review is promoted as a more reliable way of producing a high-quality review of prior research. But there are a range of threats that can undermine the reliability and quality of such reviews. One threat is the reproducibility of automated searches. [Objectives:] To evaluate the state-of-practice of reproducible searches in secondary studies, and to consider ways to improve the reproducibility of searches. [Method:] We re-run the searches of 621 secondary studies and analyse the outcomes of those (attempted) re-runs. We use the outcomes, and our experience of re-running the searches, to propose ways to improve the reproducibility of automated searches. [Results:] With the 621 studies, more than 50% of the literal search strings (ignoring other settings) are not reusable; about 87% of the searches (e.g., with settings) cannot be repeated; and around 94% of the searches (including all elements of the search) are irreproducible. We propose guidelines for automated search, directing particular attention at the formulation of search strings. [Conclusion:] While some aspects of automated search are beyond the direct control of researchers (e.g., variations in features, constraints and performance of search engines), many aspects can be effectively managed through more careful formulation and execution of the search strings themselves, and of the search settings. While the results of our evaluation are disappointing there are many simple, concrete steps that researchers can make to improve the reproducibility of their searches.