Scaling drug indication curation through crowdsourcing
Scaling drug indication curation through crowdsourcing
复制标题
DOI：
10.1093/database/bav016
复制
发表时间：
2015-03-22
影响因子：
5.8
通讯作者：
Lu, Zhiyong
中科院分区：
文献类型：
--
作者：
Khare, Ritu;Burger, John D.;Lu, Zhiyong
由于人工管理生物数据库的成本很高，人们越来越有兴趣使用计算方法来帮助人类管理员并加速人工管理过程。为了实现从FDA药物标签中对药物适应症进行编目的目标，我们最近开发了LabeledIn，这是一种针对250种临床药物的人类药物适应症资源。尽管使用了定义良好的注释指南，但它的开发需要在20周内花费超过40小时的人力。在这项研究中，我们的目标是调查通过众包技术扩展药物适应症注释的可行性，其中可以通过Amazon Mechanical Turk（MTurk）的技术环境招募未知的工人网络。为了将编目适应症的专家策展任务转化为适合MTurk上的普通工作人员的人类智能任务（HIT），我们首先简化了复杂的任务，使得每个HIT只涉及一个工作人员对给定药物标签背景下突出显示的疾病是否是适应症进行二元判断。此外，这项研究是新颖的众包界面设计中的注释准则编码到用户选项。为了评估，我们评估我们提出的方法以具有时间效率和成本效益的方式实现高质量注释的能力。我们在MTurk上发布了从706个药物标签中提取的3000多个HIT。在发布后的8小时内，我们从74名工人那里收集了18775个判断，并在450个对照HIT（已知黄金标准答案）上实现了96%的总准确率，每个药物标签的成本为1.75美元。在这些结果的基础上，我们得出结论，我们的众包方法不仅节省了大量的成本和时间，而且还导致了与领域专家相当的准确性。
Motivated by the high cost of human curation of biological databases, there is an increasing interest in using computational approaches to assist human curators and accelerate the manual curation process. Towards the goal of cataloging drug indications from FDA drug labels, we recently developed LabeledIn, a human-curated drug indication resource for 250 clinical drugs. Its development required over 40 h of human effort across 20 weeks, despite using well-defined annotation guidelines. In this study, we aim to investigate the feasibility of scaling drug indication annotation through a crowdsourcing technique where an unknown network of workers can be recruited through the technical environment of Amazon Mechanical Turk (MTurk). To translate the expert-curation task of cataloging indications into human intelligence tasks (HITs) suitable for the average workers on MTurk, we first simplify the complex task such that each HIT only involves a worker making a binary judgment of whether a highlighted disease, in context of a given drug label, is an indication. In addition, this study is novel in the crowdsourcing interface design where the annotation guidelines are encoded into user options. For evaluation, we assess the ability of our proposed method to achieve high-quality annotations in a time-efficient and cost-effective manner. We posted over 3000 HITs drawn from 706 drug labels on MTurk. Within 8 h of posting, we collected 18 775 judgments from 74 workers, and achieved an aggregated accuracy of 96% on 450 control HITs (where gold-standard answers are known), at a cost of $1.75 per drug label. On the basis of these results, we conclude that our crowdsourcing approach not only results in significant cost and time saving, but also leads to accuracy comparable to that of domain experts.