RobotReviewer: development and evaluation of a machine learning tool to speed up evidence synthesis in cardiovascular diseases

我是一名学术型的GP，其研究背景是系统综述。我之前在系统评价期刊BMJ Clinical Evidence担任临床编辑，即将提交我的博士论文，该论文探讨如何最好地与患者沟通有关心血管疾病（心脏病发作，中风和大血管疾病）的医学研究。由于对计算机编程和统计学的浓厚兴趣和能力，我正在申请信息学技能发展奖学金。该奖学金旨在进一步开发一个名为RobotReviewer的计算机系统，该系统是我与美国和荷兰的研究人员合作开发的。RobotReviewer支持研究人员进行称为系统综述的类型研究。系统综述是旨在总结特定主题的所有相关研究的文章。系统评价对临床医生特别有帮助，因为要掌握大量已发表的研究成果是不现实的。系统性综述对研究提供了一个平衡的观点，突出了大型和高质量的研究，并检查研究是否存在偏倚的风险。制作系统性综述是费力的，平均需要团队2年。现有的审查中有很大一部分已经过时，许多迫切需要的专题的审查还没有。心血管疾病尤其如此，其审查通常在2年内过期。因此，迫切需要加快编制系统性综述的技术，数据提取是编制系统性综述的一项关键（和耗时）任务，也是本提案的重点。在这里，人类评审员从他们打算总结的所有研究（通常是一大堆纸质文件）开始，并确定详细信息，包括试验如何进行和统计结果。然后将这些数据输入为审查设计的标准模板。RobotReviewer的工作方式是将现有的大量系统性综述以及描述临床试验的文章（PDF格式）纳入一个大型图书馆。RobotReviewer能够获取这些数据，并学习如何识别临床试验报告中的关键信息，这些信息是生成系统性综述所需的。到目前为止，RobotReviewer能够帮助评估临床试验是否存在偏倚风险。我们将RobotReviewer在这项任务上的准确性与人类研究人员进行了比较，发现RobotReviewer在找到讨论偏见的文本方面同样准确。总体而言，RobotReviewer在判断试验是否存在偏倚方面的准确率为70%，而人类的准确率为77%。该研究旨在进一步发展该技术，以便RobotReview能够提取许多其他重要信息，包括临床试验参与者的描述，他们使用的治疗类型，以及关于治疗的好处和危害的数据。在技术开发出来之后，它将被测试，将其准确性与人类研究人员做同样任务的准确性进行比较。为了让这项技术尽可能广泛地使用，计算机软件将免费发布。此外，我正在与科克伦合作组织（一个国际慈善机构，是世界上领先的系统评价制作者）的关键人物合作，他们对在他们的工作中试用自动化技术感兴趣。作为奖学金的一部分，我打算接受该项目所需的统计学和计算机科学方面的专业培训。此外，我将继续与美国和荷兰的计算机科学家合作，进一步发展我的技能。

I am an academic GP, whose research background is in systematic reviewing. I previously worked as a Clinical Editor at the systematic reviews journal BMJ Clinical Evidence, and am about to submit my PhD thesis which looks at how best to communicate with patients about medical research about cardiovascular diseases (heart attack, stroke, and diseases of the large blood vessels).I am applying for the Skills Development fellowship in Informatics due to a strong interest and aptitude in computer programming and statistics. This fellowship aims to further develop a computer system called RobotReviewer, which I have developed in collaboration with researchers in the US and the Netherlands.RobotReviewer supports researchers producing a type research called a systematic review. Systematic reviews are articles which aim to summarise all relevant research on a particular topic. Systematic reviews are particularly helpful to clinicians, since it is not practical to keep on top of the vast amount of research published. Systematic reviews give a balanced view on the research, giving prominence to large and high quality studies, and examining whether research is at risk of bias.Producing systematic reviews is laborious, taking a team 2 years on average. A large proportion of existing reviews are out of date, and reviews do not yet exist for many urgently needed topics. This is particularly the case in cardiovascular diseases, where reviews are typically out of date within 2 years. Technologies to speed up the production of systematic reviews are therefore urgently needed.Data extraction is a key (and time-consuming) task in producing a systematic reviews, and the one on which this proposal focuses. Here, human reviewers start with all the research they intend to summarise (typically a large pile of paper documents), and identify detailed information including on how the trial was done and statistical results. These data are then entered into a standard template designed for the review. This task is done in duplicate to ensure high accuracy.RobotReviewer works by taking in a large library of existing systematic reviews, together with articles describing clinical trials (in PDF format). RobotReviewer is able to take this data, and learn how to identify the key pieces of information in a clinical trial report which are needed to produce a systematic review. So far, RobotReviewer is able to help with the task of assessing whether clinical trials are at risk of bias. We compared the accuracy of RobotReviewer on this task against human researchers, and found that RobotReviewer was equally accurate at finding the text which discussed bias. Overall RobotReviewer was 70% accurate at judging whether a trial was biased, compared with humans who were 77% accurate.This fellowship aims to develop the technology further, so that RobotReview is able to extract many other important pieces of information, including descriptions of the participants in clinical trials, the types of treatments they used, and data on what the benefits and harms of the treatments were.After the technology has been developed, it will be tested, comparing its accuracy against the accuracy of human researchers doing the same task.In order to get the technology as widely used as possible, the computer software will be released freely. Additionally, I am working with key people in the Cochrane Collaboration (an international charity who are the world's leading producer of systematic reviews), who are interested in piloting the use of the automation technology in their work.As part of the fellowship I intend to undertake specialist training in statistics and computer science required for the project. Additionally, I will continue to collaborate with computer scientists in the US and the Netherlands on the project to develop my skills further.