Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources
Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources
复制标题
DOI：
10.1002/jrsm.1378
复制
发表时间：
2020-01-28
影响因子：
9.8
通讯作者：
Haddaway, Neal R.
中科院分区：
文献类型：
--
作者：
Gusenbauer, Michael;Haddaway, Neal R.
严格的证据识别对于系统性综述和荟萃分析（证据综合）至关重要，因为相关研究的样本选择决定了综述的结果、有效性和解释力。然而，允许访问这些证据的搜索系统提供了不同程度的精确度，召回率和再现性，并且还需要不同程度的努力。迄今为止，仍不清楚哪些搜索系统最适合证据合成以及为什么。关于选择哪些搜索引擎和书目数据库进行系统搜索的建议有限，缺乏系统的经验性绩效评估。本研究调查和比较了28个广泛使用的学术搜索系统的系统搜索质量，包括Google Scholar，PubMed和Web of Science。一种新颖的基于查询的方法测试用户与每个系统交互和检索记录的能力。该研究首次展示了搜索系统在精度、召回率和再现性方面能够有效和高效地执行（布尔）搜索的程度。我们发现搜索系统的性能存在很大差异，这意味着它们在系统搜索中的可用性各不相同。事实上，只有一半的搜索系统分析，只有少数开放存取数据库可以推荐证据合成，而不增加实质性的警告。特别是，我们的研究结果证明了为什么谷歌学术不适合作为主要的搜索系统。我们呼吁数据库所有者认识到证据综合的要求，并呼吁学术期刊重新评估系统评价的质量要求。我们的研究结果旨在支持研究人员进行更好的搜索，以获得更好的证据合成。
Rigorous evidence identification is essential for systematic reviews and meta-analyses (evidence syntheses) because the sample selection of relevant studies determines a review's outcome, validity, and explanatory power. Yet, the search systems allowing access to this evidence provide varying levels of precision, recall, and reproducibility and also demand different levels of effort. To date, it remains unclear which search systems are most appropriate for evidence synthesis and why. Advice on which search engines and bibliographic databases to choose for systematic searches is limited and lacking systematic, empirical performance assessments. This study investigates and compares the systematic search qualities of 28 widely used academic search systems, including Google Scholar, PubMed, and Web of Science. A novel, query-based method tests how well users are able to interact and retrieve records with each system. The study is the first to show the extent to which search systems can effectively and efficiently perform (Boolean) searches with regards to precision, recall, and reproducibility. We found substantial differences in the performance of search systems, meaning that their usability in systematic searches varies. Indeed, only half of the search systems analyzed and only a few Open Access databases can be recommended for evidence syntheses without adding substantial caveats. Particularly, our findings demonstrate why Google Scholar is inappropriate as principal search system. We call for database owners to recognize the requirements of evidence synthesis and for academic journals to reassess quality requirements for systematic reviews. Our findings aim to support researchers in conducting better searches for better evidence synthesis.