RobotReviewer: development and evaluation of a machine learning tool to speed up evidence synthesis in cardiovascular diseases

点击翻译按钮获取中文摘要

I am an academic GP, whose research background is in systematic reviewing. I previously worked as a Clinical Editor at the systematic reviews journal BMJ Clinical Evidence, and am about to submit my PhD thesis which looks at how best to communicate with patients about medical research about cardiovascular diseases (heart attack, stroke, and diseases of the large blood vessels).I am applying for the Skills Development fellowship in Informatics due to a strong interest and aptitude in computer programming and statistics. This fellowship aims to further develop a computer system called RobotReviewer, which I have developed in collaboration with researchers in the US and the Netherlands.RobotReviewer supports researchers producing a type research called a systematic review. Systematic reviews are articles which aim to summarise all relevant research on a particular topic. Systematic reviews are particularly helpful to clinicians, since it is not practical to keep on top of the vast amount of research published. Systematic reviews give a balanced view on the research, giving prominence to large and high quality studies, and examining whether research is at risk of bias.Producing systematic reviews is laborious, taking a team 2 years on average. A large proportion of existing reviews are out of date, and reviews do not yet exist for many urgently needed topics. This is particularly the case in cardiovascular diseases, where reviews are typically out of date within 2 years. Technologies to speed up the production of systematic reviews are therefore urgently needed.Data extraction is a key (and time-consuming) task in producing a systematic reviews, and the one on which this proposal focuses. Here, human reviewers start with all the research they intend to summarise (typically a large pile of paper documents), and identify detailed information including on how the trial was done and statistical results. These data are then entered into a standard template designed for the review. This task is done in duplicate to ensure high accuracy.RobotReviewer works by taking in a large library of existing systematic reviews, together with articles describing clinical trials (in PDF format). RobotReviewer is able to take this data, and learn how to identify the key pieces of information in a clinical trial report which are needed to produce a systematic review. So far, RobotReviewer is able to help with the task of assessing whether clinical trials are at risk of bias. We compared the accuracy of RobotReviewer on this task against human researchers, and found that RobotReviewer was equally accurate at finding the text which discussed bias. Overall RobotReviewer was 70% accurate at judging whether a trial was biased, compared with humans who were 77% accurate.This fellowship aims to develop the technology further, so that RobotReview is able to extract many other important pieces of information, including descriptions of the participants in clinical trials, the types of treatments they used, and data on what the benefits and harms of the treatments were.After the technology has been developed, it will be tested, comparing its accuracy against the accuracy of human researchers doing the same task.In order to get the technology as widely used as possible, the computer software will be released freely. Additionally, I am working with key people in the Cochrane Collaboration (an international charity who are the world's leading producer of systematic reviews), who are interested in piloting the use of the automation technology in their work.As part of the fellowship I intend to undertake specialist training in statistics and computer science required for the project. Additionally, I will continue to collaborate with computer scientists in the US and the Netherlands on the project to develop my skills further.