Optimal Stopping and Worker Selection in Crowdsourcing: an Adaptive Sequential Probability Ratio Test Framework
Optimal Stopping and Worker Selection in Crowdsourcing: an Adaptive Sequential Probability Ratio Test Framework
复制标题
DOI：
10.5705/ss.202018.0300
复制
发表时间：
2017-08
影响因子：
1.4
通讯作者：
Xiaoou Li;Yunxiao Chen;Xi Chen;Jingchen Liu;Z. Ying
中科院分区：
文献类型：
--
作者：
Xiaoou Li;Yunxiao Chen;Xi Chen;Jingchen Liu;Z. Ying
在本文中，我们的目标是在贝叶斯顺序决策框架下解决一类多重测试问题。我们的激励应用来自众包中的二元标签任务，其中请求者需要同时决定选择哪个工人提供标签以及在一定的预算限制下何时停止收集标签。我们从二元假设检验问题开始，确定单个对象的真实标签，并通过将其投射到自适应序列概率比测试（Ada-SPRT）框架下来提供最佳解决方案。我们描述了最优解的结构，即最优自适应序列设计，它通过对数似然比统计来最小化贝叶斯风险。我们还开发了一种动态规划算法，可以有效地逼近最优解。对于多重测试问题，我们进一步建议采用经验贝叶斯方法来估计类先验，并表明我们的方法具有在真实模型下收敛到最小贝叶斯风险的平均损失。模拟数据和真实数据的实验表明，与最近提出的其他几种方法相比，我们的方法具有鲁棒性，并且在标记准确性方面具有优越性。
In this paper, we aim at solving a class of multiple testing problems under the Bayesian sequential decision framework. Our motivating application comes from binary labeling tasks in crowdsourcing, where the requestor needs to simultaneously decide which worker to choose to provide the label and when to stop collecting labels under a certain budget constraint. We start with the binary hypothesis testing problem to determine the true label of a single object, and provide an optimal solution by casting it under the adaptive sequential probability ratio test (Ada-SPRT) framework. We characterize the structure of the optimal solution, i.e., optimal adaptive sequential design, which minimizes the Bayes risk through log-likelihood ratio statistic. We also develop a dynamic programming algorithm that can efficiently approximate the optimal solution. For the multiple testing problem, we further propose to adopt an empirical Bayes approach for estimating class priors and show that our method has an averaged loss that converges to the minimal Bayes risk under the true model. The experiments on both simulated and real data show the robustness of our method and its superiority in labeling accuracy as compared to several other recently proposed approaches.