Effects of Increasing Working Opportunity on Result Quality in Labor-Intensive Crowdsourcing
Effects of Increasing Working Opportunity on Result Quality in Labor-Intensive Crowdsourcing
复制标题
劳动密集型众包中工作机会增加对结果质量的影响
DOI：
10.1007/978-3-031-28035-1_19
复制
发表时间：
2023
期刊：
影响因子：
--
通讯作者：
Morishima Atsuyuki
中科院分区：
文献类型：
--
作者：
Negishi Kanta;Ito Hiroyoshi;Matsubara Masaki;Morishima Atsuyuki
在微任务众包平台中选择工人时，请求者的一种常见做法是通过查看过去任务的评估结果或通过对任务进行资格测试来选择合格的工人。这有时会错过那些可能能够完成某些任务的工人。为这类工人增加工作机会不仅对工人有利，而且对请求者也有好处，因为他们获得了劳动力资源，可以更快地完成任务。不过，一般来说，增加工作机会和获得高质量的任务结果是一种取舍；如果他们选择技能水平高于较低门槛的工人来增加工人数量，任务的质量就会受到影响。在本文中，我们通过探索不同的任务分配策略来解决劳动密集型众包中的权衡问题。为此，我们应用项目反应理论来评估员工的技能和任务的难度，并设计了一种分配任务的算法，使得员工之间分配的任务数量的差异最小，试图利用众包的潜在并行性。其次，我们提前解决了任务难度未知的问题。我们探索了一种使用ML输出进行难度估计的方法。本文报告了我们的实验结果，它显示了这种方法的潜力，并讨论了这种方法何时有效。
When selecting workers in microtask crowdsourcing platforms, a common practice of requesters is to select qualified workers by looking at the evaluation results for the tasks in the past or by conducting qualifying tests for the tasks. This sometimes misses workers who may be able to complete some of the tasks. Increasing working opportunities for such workers has advantages not only for the workers but also for requesters because they obtain labor resources for faster completion of tasks. However, in general, an increase in the working opportunity and obtaining high-quality task results is a trade-off; if they choose workers whose skill levels are above a lower threshold to increase the number of workers, the quality of the task will be undermined. In this paper, we address the problem of improving the trade-off in labor-intensive crowdsourcing by exploring different task assignment strategies. For that purpose, we apply Item Response Theory to evaluate the skills of workers and the difficulty of tasks and devise an algorithm for assigning tasks in such a way that the variance in the number of tasks assigned among workers is minimized trying to take advantage of the potential parallelism of crowdsourcing. Second, we address the problem that the difficulty of the tasks is unknown in advance. We explore an approach that uses ML outputs for difficulty estimation. This paper reports on our experimental result, which shows the potential of this approach, and discusses when this approach is effective.