RobotReviewer: development and evaluation of a machine learning tool to speed up evidence synthesis in cardiovascular diseases

我是一名学术全科医生，研究背景是系统综述。我之前曾在系统综述杂志《BMJ临床证据》担任临床编辑，即将提交我的博士论文，该论文着眼于如何最好地与患者沟通有关心血管疾病(心脏病发作、中风和大血管疾病)的医学研究。由于对计算机编程和统计学有浓厚的兴趣和天赋，我正在申请信息学技能发展奖学金。这项奖学金旨在进一步开发一种名为RobotReviewer的计算机系统，该系统是我与美国和荷兰的研究人员合作开发的。RobotReviewer支持研究人员进行一种名为系统审查的类型研究。系统综述是旨在总结某一特定主题的所有相关研究的文章。系统评价对临床医生特别有帮助，因为保持对已发表的大量研究的了解是不现实的。系统评价对研究提供了一个平衡的观点，突出大型和高质量的研究，并检查研究是否存在偏见的风险。产生系统评价是费力的，一个团队平均需要2年时间。现有的审查有很大一部分已经过时，许多迫切需要的主题还没有审查。尤其是在心血管疾病方面，审查通常在两年内过时。因此，迫切需要加快产生系统评价的技术。数据提取是产生系统评价的一项关键(和耗时)任务，也是本提案的重点之一。在这里，人类审查者从他们打算总结的所有研究(通常是一大堆纸质文件)开始，并确定详细的信息，包括试验是如何进行的和统计结果。然后将这些数据输入为审查而设计的标准模板。这项任务是一式两份完成的，以确保高准确性。RobotReviewer的工作方式是将现有的系统评价以及描述临床试验的文章(PDF格式)纳入大型库中。RobotReviewer能够获取这些数据，并学习如何识别临床试验报告中的关键信息，这些信息是产生系统评价所需的。到目前为止，RobotReview能够帮助完成评估临床试验是否存在偏差风险的任务。我们比较了机器人审查者和人类研究人员在这项任务上的准确性，发现机器人审查者在找到讨论偏见的文本方面同样准确。总体而言，RobotReview在判断试验是否存在偏见方面的准确率为70%，而人类的准确率为77%。该奖学金旨在进一步开发这项技术，使RobotReview能够提取许多其他重要信息，包括临床试验参与者的描述、他们使用的治疗类型，以及治疗的益处和危害的数据。在开发出这项技术后，将对其进行测试，将其准确性与人类研究人员执行相同任务的准确性进行比较。为了尽可能广泛地应用该技术，计算机软件将免费发布。此外，我正在与Cochrane Collaboration(一家国际慈善机构，是世界领先的系统评估生产者)的关键人物合作，他们对在他们的工作中试验使用自动化技术感兴趣。作为奖学金的一部分，我打算接受该项目所需的统计学和计算机科学方面的专业培训。此外，我将继续与美国和荷兰的计算机科学家在该项目上合作，以进一步发展我的技能。

I am an academic GP, whose research background is in systematic reviewing. I previously worked as a Clinical Editor at the systematic reviews journal BMJ Clinical Evidence, and am about to submit my PhD thesis which looks at how best to communicate with patients about medical research about cardiovascular diseases (heart attack, stroke, and diseases of the large blood vessels).I am applying for the Skills Development fellowship in Informatics due to a strong interest and aptitude in computer programming and statistics. This fellowship aims to further develop a computer system called RobotReviewer, which I have developed in collaboration with researchers in the US and the Netherlands.RobotReviewer supports researchers producing a type research called a systematic review. Systematic reviews are articles which aim to summarise all relevant research on a particular topic. Systematic reviews are particularly helpful to clinicians, since it is not practical to keep on top of the vast amount of research published. Systematic reviews give a balanced view on the research, giving prominence to large and high quality studies, and examining whether research is at risk of bias.Producing systematic reviews is laborious, taking a team 2 years on average. A large proportion of existing reviews are out of date, and reviews do not yet exist for many urgently needed topics. This is particularly the case in cardiovascular diseases, where reviews are typically out of date within 2 years. Technologies to speed up the production of systematic reviews are therefore urgently needed.Data extraction is a key (and time-consuming) task in producing a systematic reviews, and the one on which this proposal focuses. Here, human reviewers start with all the research they intend to summarise (typically a large pile of paper documents), and identify detailed information including on how the trial was done and statistical results. These data are then entered into a standard template designed for the review. This task is done in duplicate to ensure high accuracy.RobotReviewer works by taking in a large library of existing systematic reviews, together with articles describing clinical trials (in PDF format). RobotReviewer is able to take this data, and learn how to identify the key pieces of information in a clinical trial report which are needed to produce a systematic review. So far, RobotReviewer is able to help with the task of assessing whether clinical trials are at risk of bias. We compared the accuracy of RobotReviewer on this task against human researchers, and found that RobotReviewer was equally accurate at finding the text which discussed bias. Overall RobotReviewer was 70% accurate at judging whether a trial was biased, compared with humans who were 77% accurate.This fellowship aims to develop the technology further, so that RobotReview is able to extract many other important pieces of information, including descriptions of the participants in clinical trials, the types of treatments they used, and data on what the benefits and harms of the treatments were.After the technology has been developed, it will be tested, comparing its accuracy against the accuracy of human researchers doing the same task.In order to get the technology as widely used as possible, the computer software will be released freely. Additionally, I am working with key people in the Cochrane Collaboration (an international charity who are the world's leading producer of systematic reviews), who are interested in piloting the use of the automation technology in their work.As part of the fellowship I intend to undertake specialist training in statistics and computer science required for the project. Additionally, I will continue to collaborate with computer scientists in the US and the Netherlands on the project to develop my skills further.