MicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy
MicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy
复制标题
DOI：
10.1609/hcomp.v4i1.13270
复制
发表时间：
2016-09
期刊：
影响因子：
--
通讯作者：
Ryan Drapeau;Lydia B. Chilton;Jonathan Bragg;Daniel S. Weld
中科院分区：
文献类型：
--
作者：
Ryan Drapeau;Lydia B. Chilton;Jonathan Bragg;Daniel S. Weld
人群工作者是人，因此有时会犯错误。为了确保最高质量的输出，请求者经常发布冗余作业，这些作业具有黄金测试问题和基于期望最大化（EM）的复杂聚合机制。虽然这些方法在许多情况下都能得到准确的结果，但它们在具有局部最小值的极端困难问题上失败了，例如大多数工人得到错误答案的情况。事实上，这已经导致一些研究人员得出结论，在某些任务上，众包永远无法达到高精度，无论有多少工人参与。本文提出了一种新的质量控制工作流程，称为MicroTalk，它要求一些工人证明他们的推理，并要求其他人重新考虑他们的决定后，阅读反对意见的工人。Amazon Mechanical Turk的工作人员在一项具有挑战性的NLP注释任务上进行的实验表明，（1）论证将单个工作人员的准确性提高了20%，（2）将考虑限制在具有复杂解释的工作人员身上，可以进一步提高准确性，（3）对于一系列预算，我们完整的MicroTalk聚合工作流程比简单的投票方法产生更高的准确性。
Crowd workers are human and thus sometimes make mistakes. In order to ensure the highest quality output, requesters often issue redundant jobs with gold test questions and sophisticated aggregation mechanisms based on expectation maximization (EM). While these methods yield accurate results in many cases, they fail on extremely difficult problems with local minima, such as situations where the majority of workers get the answer wrong. Indeed, this has caused some researchers to conclude that on some tasks crowdsourcing can never achieve high accuracies, no matter how many workers are involved. This paper presents a new quality-control workflow, called MicroTalk, that requires some workers to Justify their reasoning and asks others to Reconsider their decisions after reading counter-arguments from workers with opposing views. Experiments on a challenging NLP annotation task with workers from Amazon Mechanical Turk show that (1) argumentation improves the accuracy of individual workers by 20%, (2) restricting consideration to workers with complex explanations improves accuracy even more, and (3) our complete MicroTalk aggregation workflow produces much higher accuracy than simpler voting approaches for a range of budgets.