Efficient Online Scalar Annotation with Bounded Support
Efficient Online Scalar Annotation with Bounded Support
复制标题
DOI：
10.18653/v1/p18-1020
复制
发表时间：
2018-06
期刊：
影响因子：
--
通讯作者：
Keisuke Sakaguchi;Benjamin Van Durme
中科院分区：
文献类型：
--
作者：
Keisuke Sakaguchi;Benjamin Van Durme
我们描述了一种新的方法，可以有效地获取标量注释，用于构建数据集和通过人的判断进行系统质量评估。我们对比了直接评估(注释者直接给条目赋分)、在线成对排名聚合(分数来自注释者对条目的比较)和本文提出的混合方法(EASL：Efficient Annotation of Scalar Label)。我们的建议导致了与地面事实的相关性增加，注释器的效率要高得多，这表明这一策略是一种改进的数据集创建和手动系统评估机制。
We describe a novel method for efficiently eliciting scalar annotations for dataset construction and system quality estimation by human judgments. We contrast direct assessment (annotators assign scores to items directly), online pairwise ranking aggregation (scores derive from annotator comparison of items), and a hybrid approach (EASL: Efficient Annotation of Scalar Labels) proposed here. Our proposal leads to increased correlation with ground truth, at far greater annotator efficiency, suggesting this strategy as an improved mechanism for dataset creation and manual system evaluation.