Hybrid Approaches to Optimizing Evidence Synthesis via Machine Learning and Crowdsourcing

摘要 系统评价构成了最高质量的证据， 循证医学（Evidence-Based Medicine，EBM）如今，这样的审查可以告知一切 从国家卫生政策指南到床边护理。然而，系统评价 生产极其费力;研究人员再也无法跟上大规模的 目前正在公布的证据。 通过机器学习（ML）实现系统综述生产的半自动化， 证明了大幅减少评审员工作量的潜力，同时保持 全面发展然而，机器不太可能完全取代人类。 评论家在不久的将来相反，人类专家可能会留在循环中， 通过自动化的方法。利用人类工作者交集的方法 系统评价背景下的ML模型尚未详细探讨。 此外，我们认为，在利用分布式 众工为系统评价做出贡献，从而节省专家评审员 努力这种新颖的途径在很大程度上被忽视，作为增加 审查生产效率。 我们建议通过开发和评估新的，混合的， 联合领域专家进行系统评价的方法 （系统审查员），通过众包平台招募的外行工人， 亚马逊的土耳其机器人和志愿公民科学家，同时 利用ML模型。 这种创新的混合方法将是智能化的首次深入探索， ML/人类系统，旨在减少生物医学生产中的工作量 系统评价我们强有力的初步工作证明了这一点的承诺 总体战略。

Abstract Systematic reviews constitute the highest quality of evidence and form the cornerstone of evidence-based medicine (EBM). Such reviews now inform everything from national health policy guidelines to bedside care. However, systematic reviews are extremely laborious to produce; researchers can no longer keep pace with the massive amount of evidence now being published. Semi-automation of systematic review production via machine learning (ML) has demonstrated the potential to substantially reduce reviewer workload while maintaining comprehensiveness. However, it is unlikely that machines will fully supplant human reviewers in the near future. Rather, human experts will probably remain in the loop, assisted by automated methods. Methods that exploit the intersection of human workers and ML models in the context of systematic reviews have not been explored at length. Furthermore, we believe there is substantial untapped potential in harnessing distributed crowd-workers to contribute to systematic reviews, and thus economize expert reviewer efforts. This novel avenue has largely been neglected as a means of increasing the efficiency of review production. We propose addressing this gap by developing and evaluating novel, hybrid approaches to generating systematic reviews that jointly incorporate domain experts (systematic reviewers), layperson workers recruited via crowdworking platforms such as Amazon's Mechanical Turk and volunteer citizen scientists, while simultaneously capitalizing on ML models. This innovative, hybrid approach will be the first in-depth exploration of intelligent ML/human systems that aim to reduce the workload in the production of biomedical systematic reviews. Our strong preliminary work demonstrates the promise of this general strategy.