Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to Ensure Quality Relevance Annotations
Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to Ensure Quality Relevance Annotations
复制标题
您的行为表明您的可靠性：对人群行为轨迹进行建模以确保质量相关注释
DOI：
10.1609/hcomp.v6i1.13331
复制
发表时间：
2018
期刊：
影响因子：
--
通讯作者：
Matthew Lease
中科院分区：
文献类型：
--
作者：
Tanya Goyal;Tyler McDonnell;Mucahid Kutlu;T. Elsayed;Matthew Lease
虽然同行协议和黄金检查是确保众包数据收集质量的成熟方法，但我们探索了一个相对较新的质量控制方向：直接从注释期间收集的工人行为痕迹中估计工作质量。我们提出了三个基于行为的模型来预测标签的正确性和工人的准确性，然后进一步将模型预测应用于标签聚合和标签收集的优化。作为这项工作的一部分，我们收集并分享了一个新的Mechanical Turk行为信号数据集，用于判断搜索结果的相关性。结果表明，行为数据可以有效地用于预测工作质量，这可能是特别有用的单一标签或在冷启动的情况下，个人的先前的工作历史是不可用的。我们进一步展示了标签聚合的改进，并在确保数据质量的同时降低了标签成本。
While peer-agreement and gold checks are well-established methods for ensuring quality in crowdsourced data collection, we explore a relatively new direction for quality control: estimating work quality directly from workers’ behavioral traces collected during annotation. We propose three behavior-based models to predict label correctness and worker accuracy, then further apply model predictions to label aggregation and optimization of label collection. As part of this work, we collect and share a new Mechanical Turk dataset of behavioral signals judging the relevance of search results. Results show that behavioral data can be effectively used to predict work quality, which could be especially useful with single labeling or in a cold start scenario in which individuals’ prior work history is unavailable. We further show improvement in label aggregation and reducing labeling cost while ensuring data quality.