Effect of written instructions on assessor agreement

Effect of written instructions on assessor agreement
复制标题

书面指示对评估员协议的影响

DOI：
--
复制
发表时间：
2012
期刊：
Annual International ACM SIGIR Conference on Research and Development in Information Retrieval
影响因子：
--
通讯作者：
Marjorie Desamito
Marjorie Desamito
中科院分区：
--
文献类型：
--
作者：
William Webber;Bryan N. Toth;Marjorie Desamito

文献摘要

被引文献

相似文献

Assessors frequently disagree on the topical relevance of documents. How much of this disagreement is due to ambiguity in assessment instructions? We have two assessors assess TREC Legal Track documents for relevance, some to a general topic description, others to detailed assessment guidelines. We find that detailed guidelines lead to no significant increase in agreement amongst assessors or between assessors and the official qrels.