Identifying relevant studies for systematic reviews and health technology assessments using text mining

系统评价是一种广泛使用的方法，用于以可靠的方式汇集多项研究的结果，并经常用于为政策和实践提供信息(如制定指南)。系统评价的一个关键特征是应用科学方法来发现并尽量减少研究选择和处理中的偏见和错误。然而，大量和不断增长的已发表研究及其发表速度的增加，使得以一种公正的方式识别相关研究的任务变得复杂和耗时。审查者经常需要手动查看数千个不相关的标题和摘要，以确定数量少得多的相关标题和摘要；这一过程被称为“筛选”。考虑到一位经验丰富的审查者可能需要30秒到几分钟来评估一篇引文，筛选1万篇引文所涉及的工作是相当可观的(筛选的负担有时比这高得多)。节省评论时间的明显方法是简单地筛选更少的研究。目前，这通常是通过制定更具体的搜索战略来减少通过电子搜索检索到的引文数量，从而减少找到的无关引文数量来实现的。然而，限制搜索的敏感性可能会破坏系统审查的最重要原则之一：其结果基于一组不偏不倚的研究。因此，我们建议开发和评估一种解决这两个问题的替代方法：尽可能地进行敏感搜索很重要，因为这是获得可靠审查结果所必需的；但有时也不可能筛选这些敏感搜索将产生的引文数量。因此，需要某种形式的自动化来识别哪些引文需要人工筛选，哪些不需要人工筛选。由于自动化必须处理的数据是文本的形式，我们正在期待相对较新的文本挖掘科学来为这些问题提供解决方案。有两种使用文本挖掘的方法特别有希望在系统审查中帮助筛选：一种方法旨在对人工筛选的项目列表进行优先排序，以便列表顶部的研究是最有可能相关的研究(筛选优先级)；第二种方法使用手动分配的包括/排除研究类别，以便“学习”自动应用此类分类(“自动分类”)。我们知道目前还没有对筛选优先级的评估。还有少数其他小组正在开发自动分类工具，但这个项目通过以下方式增加价值：在持续审查中实施技术；为他们使用这类审查制定衡量标准；与系统审查人员和计算机科学家接触，以期建设进一步实施和发展的能力。由于这些技术的使用和为其使用开发有效方法尚处于初级阶段，该项目的一个重要部分是推广：培养对未来使用这些技术的兴趣、能力和热情。通过减少审查中筛选的负担，使用文本挖掘的新方法可能使对两者的系统审查：更快地完成(从而满足严格的政策和实践时间表，并提高其成本效益)；最大限度地减少发表偏见的影响，减少错过相关研究的机会(使他们能够提高搜索的敏感性)。反过来，通过促进更及时和更可靠的审查，这种方法有可能改善整个卫生部门和其他部门的决策。

Systematic reviews are a widely used method to bring together the findings from multiple studies in a reliable way, and are often used to inform policy and practice (such as guideline development). A critical feature of a systematic review is the application of scientific method to uncover and minimise bias and error in the selection and treatment of studies. However, the large and growing number of published studies, and their increasing rate of publication, makes the task of identifying relevant studies in an unbiased way both complex and time consuming.Unfortunately, the specificity of sensitive electronic searches of bibliographic databases is low. Reviewers often need to look manually through many thousands of irrelevant titles and abstracts in order to identify the much smaller number of relevant ones; a process known as 'screening'. Given that an experienced reviewer can take between 30 seconds and several minutes to evaluate a citation, the work involved in screening 10,000 citations is considerable (and the burden of screening is sometimes considerably higher than this).The obvious way to save time in reviews is simply to screen fewer studies. Currently, this is usually accomplished by reducing the number of citations retrieved through electronic searches by developing more specific search strategies, thereby reducing the number of irrelevant citations found. However, limiting the sensitivity of a search may undermine one of the most important principles of a systematic review: that its results are based on an unbiased set of studies.We therefore propose to develop and evaluate an alternative approach which addresses both of these issues: it is important to have as sensitive a search as is possible, as this is necessary to obtain reliable review findings; but it is also sometimes impossible to screen the number of citations that these sensitive searches will generate. Thus, some form of automation is needed to identify the citations that do, and do not, need to be screened manually. As the data upon which the automation must work are in the form of text, we are looking to the relatively new science of text mining to provide solutions to these problems.There are two ways of using text mining that are particularly promising for assisting with screening in systematic reviews: one aims to prioritise the list of items for manual screening so that the studies at the top of the list are those that are most likely to be relevant ('screening prioritisation'); the second method uses the manually assigned include/exclude categories of studies in order to 'learn' to apply such categorisations automatically ('automatic classification').We know of no existing evaluations of screening prioritisation. There are a small number of other groups developing tools for automatic classification, but this project adds value by: implementing the technology in ongoing reviews; developing metrics for their use such reviews; and engaging with systematic reviewers and computer scientists with a view to building capacity for further implementation and development. As the use of these technologies and the development of validated methods for their use are in their infancy, an important part of the project is outreach: to build interest, capacity and enthusiasm for their use in the future. By reducing the burden of screening in reviews, new methodologies using text mining may enable systematic reviews to both: be completed more quickly (thus meeting exacting policy and practice timescales and increasing their cost efficiency); AND minimise the impact of publication bias and reduce the chances that relevant research will be missed (by enabling them to increase the sensitivity of their searches). In turn, by facilitating more timely and reliable reviews, this methodology has the potential to improve decision-making across the health sector and beyond.