Identifying relevant studies for systematic reviews and health technology assessments using text mining

点击翻译按钮获取中文摘要

Systematic reviews are a widely used method to bring together the findings from multiple studies in a reliable way, and are often used to inform policy and practice (such as guideline development). A critical feature of a systematic review is the application of scientific method to uncover and minimise bias and error in the selection and treatment of studies. However, the large and growing number of published studies, and their increasing rate of publication, makes the task of identifying relevant studies in an unbiased way both complex and time consuming.Unfortunately, the specificity of sensitive electronic searches of bibliographic databases is low. Reviewers often need to look manually through many thousands of irrelevant titles and abstracts in order to identify the much smaller number of relevant ones; a process known as 'screening'. Given that an experienced reviewer can take between 30 seconds and several minutes to evaluate a citation, the work involved in screening 10,000 citations is considerable (and the burden of screening is sometimes considerably higher than this).The obvious way to save time in reviews is simply to screen fewer studies. Currently, this is usually accomplished by reducing the number of citations retrieved through electronic searches by developing more specific search strategies, thereby reducing the number of irrelevant citations found. However, limiting the sensitivity of a search may undermine one of the most important principles of a systematic review: that its results are based on an unbiased set of studies.We therefore propose to develop and evaluate an alternative approach which addresses both of these issues: it is important to have as sensitive a search as is possible, as this is necessary to obtain reliable review findings; but it is also sometimes impossible to screen the number of citations that these sensitive searches will generate. Thus, some form of automation is needed to identify the citations that do, and do not, need to be screened manually. As the data upon which the automation must work are in the form of text, we are looking to the relatively new science of text mining to provide solutions to these problems.There are two ways of using text mining that are particularly promising for assisting with screening in systematic reviews: one aims to prioritise the list of items for manual screening so that the studies at the top of the list are those that are most likely to be relevant ('screening prioritisation'); the second method uses the manually assigned include/exclude categories of studies in order to 'learn' to apply such categorisations automatically ('automatic classification').We know of no existing evaluations of screening prioritisation. There are a small number of other groups developing tools for automatic classification, but this project adds value by: implementing the technology in ongoing reviews; developing metrics for their use such reviews; and engaging with systematic reviewers and computer scientists with a view to building capacity for further implementation and development. As the use of these technologies and the development of validated methods for their use are in their infancy, an important part of the project is outreach: to build interest, capacity and enthusiasm for their use in the future. By reducing the burden of screening in reviews, new methodologies using text mining may enable systematic reviews to both: be completed more quickly (thus meeting exacting policy and practice timescales and increasing their cost efficiency); AND minimise the impact of publication bias and reduce the chances that relevant research will be missed (by enabling them to increase the sensitivity of their searches). In turn, by facilitating more timely and reliable reviews, this methodology has the potential to improve decision-making across the health sector and beyond.