POMDP-based control of workflows for crowdsourcing
POMDP-based control of workflows for crowdsourcing
复制标题
DOI：
10.1016/j.artint.2013.06.002
复制
发表时间：
2013-09-01
影响因子：
14.4
通讯作者：
Weld, Daniel S.
中科院分区：
文献类型：
--
作者：
Dai, Peng;Lin, Christopher H.;Weld, Daniel S.
众包，即通过公开征集将任务外包给一群不知名的人（“工人”），正在迅速流行。它已经被众多雇主（“请求者”）大量使用来解决各种任务，例如音频转录、内容筛选和标记机器学习的训练数据。然而，由于工人质量的高度可变性，此类任务的质量控制仍然是一个关键挑战。在本文中，我们展示了决策理论技术对于优化众包中使用的工作流程问题的价值。特别是，我们设计了将贝叶斯网络学习和推理与部分可观察马尔可夫决策过程（POMDP）相结合的人工智能代理，以获得出色的成本质量权衡。我们将这些技术用于三种不同的众包场景：（1）控制投票来回答二元选择问题，（2）控制迭代改进工作流程，以及（3）控制任务的备用工作流程之间的切换。在每个场景中，我们设计了一个贝叶斯网络模型，将工人能力、任务难度和工人响应质量联系起来。我们还为每个任务设计了一个POMDP，其解决方案提供了动态控制策略。我们在 Amazon Mechanical Turk 上的现场实验中展示了我们的模型和代理的实用性。我们始终如一地获得比非自适应控制器更高的质量结果，同时产生相同或更少的成本。 (C) 2013 Elsevier B.V. 保留所有权利。
Crowdsourcing, outsourcing of tasks to a crowd of unknown people ("workers") in an open call, is rapidly rising in popularity. It is already being heavily used by numerous employers ("requesters") for solving a wide variety of tasks, such as audio transcription, content screening, and labeling training data for machine learning. However, quality control of such tasks continues to be a key challenge because of the high variability in worker quality. In this paper we show the value of decision-theoretic techniques for the problem of optimizing workflows used in crowdsourcing. In particular, we design Al agents that use Bayesian network learning and inference in combination with Partially-Observable Markov Decision Processes (POMDPs) for obtaining excellent cost-quality tradeoffs.We use these techniques for three distinct crowdsourcing scenarios: (1) control of voting to answer a binary-choice question, (2) control of an iterative improvement workflow, and (3) control of switching between alternate workflows for a task. In each scenario, we design a Bayes net model that relates worker competency, task difficulty and worker response quality. We also design a POMDP for each task, whose solution provides the dynamic control policy. We demonstrate the usefulness of our models and agents in live experiments on Amazon Mechanical Turk. We consistently achieve superior quality results than non-adaptive controllers, while incurring equal or less cost. (C) 2013 Elsevier B.V. All rights reserved.