An exploration of crowdsourcing citation screening for systematic reviews.
An exploration of crowdsourcing citation screening for systematic reviews.
复制标题
DOI：
10.1002/jrsm.1252
复制
发表时间：
2017-09
影响因子：
9.8
通讯作者：
Wallace BC
中科院分区：
文献类型：
--
作者：
Mortensen ML;Adam GP;Trikalinos TA;Kraska T;Wallace BC
系统评估越来越多地被用于为医疗保健决策提供信息，但制作成本高昂。我们探索使用众包(通过网络将任务分配给未经培训的工人)来降低筛选引文的成本。我们使用Amazon Machine Turk作为我们的平台，并以4个之前进行的系统评估为例。对于每一次引用，工作人员回答了4到5个相当于资格标准的问题。我们使用9种算法中的1种将来自多个工作人员的响应汇总到包括或排除引文的总体决策中，并将这些算法的性能与训练有素的专家的相应决策进行了比较。最具包容性的算法(如果任何工作人员这样做，则将引文指定为相关)确定了95%到99%的最终被纳入评论的引文，而排除了68%到82%的无关引文。其他算法增加了排除无关文章的比例，但要付出一些代价才能纳入相关研究。众包工作人员在4至17天内完成筛选，成本为460至2220美元，与训练有素的专家相比，成本降低高达88%。众包可能是一种有用的方法，可以降低为系统审查确定文献的成本。
Systematic reviews are increasingly used to inform health care decisions, but are expensive to produce. We explore the use of crowdsourcing (distributing tasks to untrained workers via the web) to reduce the cost of screening citations. We used Amazon Mechanical Turk as our platform and 4 previously conducted systematic reviews as examples. For each citation, workers answered 4 or 5 questions that were equivalent to the eligibility criteria. We aggregated responses from multiple workers into an overall decision to include or exclude the citation using 1 of 9 algorithms and compared the performance of these algorithms to the corresponding decisions of trained experts. The most inclusive algorithm (designating a citation as relevant if any worker did) identified 95% to 99% of the citations that were ultimately included in the reviews while excluding 68% to 82% of irrelevant citations. Other algorithms increased the fraction of irrelevant articles excluded at some cost to the inclusion of relevant studies. Crowdworkers completed screening in 4 to 17 days, costing $460 to $2220, a cost reduction of up to 88% compared to trained experts. Crowdsourcing may represent a useful approach to reducing the cost of identifying literature for systematic reviews.
登录
查看更多内容
影响因子：
16.8
作者：
Silvertown, Jonathan
通讯作者：
Silvertown, Jonathan
影响因子：
15.8
作者：
Bastian H;Glasziou P;Chalmers I
通讯作者：
Chalmers I
影响因子：
5.1
作者：
Goodchild, Michael F.;Glennon, J. Alan
通讯作者：
Glennon, J. Alan
影响因子：
22.4
作者：
FLEISS, JL
通讯作者：
FLEISS, JL
DOI：
10.1111/j.1365-2966.2008.13689.x
发表时间：
2008-09-21
影响因子：
4.8
作者：
Lintott, Chris J.;Schawinski, Kevin;Vandenberg, Jan
通讯作者：
Vandenberg, Jan