The annotation-validation (AV) model
The annotation-validation (AV) model
复制标题
注释验证（AV）模型
DOI：
10.1145/2594776.2594779
复制
发表时间：
2014
期刊：
影响因子：
--
通讯作者：
Chamberlain J
中科院分区：
文献类型：
--
作者：
Chamberlain J
评估用户对大型数据集系统的贡献是许多领域的挑战，从众包中的任务评估到信息检索中的文档相关性。本文介绍了一个模型，用于奖励和评估用户使用回顾性验证，只有一个小的金标准需要启动系统。该模型的模拟表明，用户得到适当的高质量的响应，但从文本注释游戏中的模型的实现数据的分析表明，它可能不够复杂，预测用户的表现。
Evaluating contributions from users of systems with large datasets is a challenge across many domains, from task assessment in crowdsourcing to document relevance in information retrieval. This paper introduces a model for rewarding and evaluating users using retrospective validation, with only a small gold standard required to initiate the system. A simulation of the model shows that users are rewarded appropriately for high quality responses however analysis of data from an implementation of the model in a text annotation game indicates it may not be sophisticated enough to predict user performance.