Why Is That Relevant? Collecting Annotator Rationales for Relevance Judgments
Why Is That Relevant? Collecting Annotator Rationales for Relevance Judgments
复制标题
DOI：
10.1609/hcomp.v4i1.13287
复制
发表时间：
2016-09
期刊：
影响因子：
--
通讯作者：
Tyler McDonnell;Matthew Lease;Mucahid Kutlu;T. Elsayed
中科院分区：
文献类型：
--
作者：
Tyler McDonnell;Matthew Lease;Mucahid Kutlu;T. Elsayed
在收集项目的主观人类评分时，由于任务的主观性以及缺乏对法官如何做出每个评分决定的深入了解，可能很难衡量和执行数据质量。为了解决这个问题，我们建议要求法官提供每个评级决定背后的特定类型的理由。我们在信息检索领域评估这种方法，人类法官评估网页与搜索查询的相关性。 Mechanical Turk 上收集的超过 10,000 个判断的成本效益分析表明这是双赢的：经验丰富的众包工作人员在几乎不增加任务完成时间的情况下提供了基本原理，同时提供了许多进一步的好处，包括更可靠的判断和评估人类评估者及其判断的更大透明度。进一步的好处包括减少对专家黄金的需求、评级和理由双重监督的机会以及理由本身的附加值。
When collecting subjective human ratings of items, it can be difficult to measure and enforce data quality due to task subjectivity and lack of insight into how judges’ arrive at each rating decision. To address this, we propose requiring judges to provide a specific type of rationale underlying each rating decision. We evaluate this approach in the domain of Information Retrieval, where human judges rate the relevance of Webpages to search queries. Cost-benefit analysis over 10,000 judgments collected on Mechanical Turk suggests a win-win: experienced crowd workers provide rationales with almost no increase in task completion time while providing a multitude of further benefits, including more reliable judgments and greater transparency for evaluating both human raters and their judgments. Further benefits include reduced need for expert gold, the opportunity for dual-supervision from ratings and rationales, and added value from the rationales themselves.