Implications of Inter-Rater Agreement on a Student Information Retrieval Evaluation
Implications of Inter-Rater Agreement on a Student Information Retrieval Evaluation
复制标题
评估者间协议对学生信息检索评估的影响
DOI：
--
复制
发表时间：
2010
期刊：
影响因子：
--
通讯作者：
Peter Mutschke
中科院分区：
文献类型：
--
作者：
Philipp Schaer;Philipp Mayr;Peter Mutschke
本文是关于三种不同检索支持服务的信息检索评估。所有三项服务均旨在弥补元数据驱动的数字图书馆中出现的典型问题，这些问题无法通过简单的基于 tf-idf 的检索来充分处理。这些服务是：（1）基于共词分析的查询扩展机制和通过（2）布拉德福德化和（3）作者中心性重新排名。这些服务由 73 名信息科学专业学生进行相关性评估进行评估。由于学生既不是信息专业人员也不是领域专家，因此需要考虑评估者之间的一致性问题。出现了两个重要的影响：（1）评估者间的一致性率主要是中等至中等；（2）在数据清理步骤删除了一致性率较低的评估之后，评估数据显示三个检索服务返回了不相交但仍然相关的结果集。
This paper is about an information retrieval evaluation on three different retrieval-supporting services. All three services were designed to compensate typical problems that arise in metadata-driven Digital Libraries, which are not adequately handled by a simple tf-idf based retrieval. The services are: (1) a co-word analysis based query expansion mechanism and re-ranking via (2) Bradfordizing and (3) author centrality. The services are evaluated with relevance assessments conducted by 73 information science students. Since the students are neither information professionals nor domain experts the question of inter-rater agreement is taken into consideration. Two important implications emerge: (1) the inter-rater agreement rates were mainly fair to moderate and (2) after a data-cleaning step which erased the assessments with poor agreement rates the evaluation data shows that the three retrieval services returned disjoint but still relevant result sets.