Quality Assessment of Crowdwork via Eye Gaze: Towards Adaptive Personalized Crowdsourcing
Quality Assessment of Crowdwork via Eye Gaze: Towards Adaptive Personalized Crowdsourcing
复制标题
DOI：
10.1007/978-3-030-85616-8_8
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Md. Rabiul Islam;Shun Nawa;Andrew W. Vargo;M. Iwata;Masaki Matsubara;Atsuyuki Morishima;K. Kise
中科院分区：
文献类型：
--
作者：
Md. Rabiul Islam;Shun Nawa;Andrew W. Vargo;M. Iwata;Masaki Matsubara;Atsuyuki Morishima;K. Kise
创建高效和公平的众包平台的一个重大挑战是快速评估众包工作的质量。如果众包工作者缺乏技能、动机或理解力来提供足够高质量的任务完成，这会降低平台的效率。虽然这似乎只是任务提供者的问题，但现实是，这个问题的负担越来越多地被众包工作者所利用。例如，任务提供者可能在任务结果的评估已经完成之后不向众包工作者支付他们的工作。在本文中，我们提出的方法，快速评估群众工作的质量，使用眼睛注视信息，估计正确答案率。我们发现，自监督学习（SSL）生成的功能的方法提供了最有效的结果，平均绝对误差为0.09。研究结果显示了使用眼睛注视信息来促进自适应个性化众包平台的潜力。
A significant challenge for creating efficient and fair crowdsourcing platforms is in rapid assessment of the quality of crowdwork. If a crowdworker lacks the skill, motivation, or understanding to provide adequate quality task completion, this reduces the efficacy of a platform. While this would seem like only a problem for task providers, the reality is that the burden of this problem is increasingly leveraged on crowdworkers. For example, task providers may not pay crowdworkers for their work after the evaluation of the task results has been completed. In this paper, we propose methods for quickly evaluating the quality of crowdwork using eye gaze information by estimating the correct answer rate. We find that the method with features generated by self-supervised learning (SSL) provides the most efficient result with a mean absolute error of 0.09. The results exhibit the potential of using eye gaze information to facilitate adaptive personalized crowdsourcing platforms.