Variations in Relevance Assessments and the Measurement of Retrieval Effectiveness
Variations in Relevance Assessments and the Measurement of Retrieval Effectiveness
复制标题
相关性评估和检索有效性测量的变化
DOI：
--
复制
发表时间：
1996
期刊：
影响因子：
--
通讯作者：
S. P. Harter
中科院分区：
文献类型：
--
作者：
S. P. Harter
本文的目的是提请注意相关性评估中的差异问题，以及这些差异可能对检索效果的衡量产生的影响。通过对文献的分析回顾，我表明，尽管在实验测试集合中的相关性评估存在着广泛的差异，但它们对提取性能测量的影响几乎完全没有研究过。我将进一步论证，我们所知道的在实验条件下影响相关性评估的许多变量，以及我们对心理、情景、基于用户的相关性的新理解，都指向一个单一的结论。我们不能再把对信息检索系统的评价停留在这样的假设上，即这种变化不会对信息检索性能的衡量产生重大影响。需要进行一系列彻底、严格和广泛的测试，以准确地确定相关性评估的变化如何以及在什么条件下影响检索性能的衡量标准。我们需要制定对这些变化以及更普遍地对人为因素和个人差异敏感的评估方法。我们的评估方法必须反映真实用户的真实世界。©1996 John Wiley父子公司
The purpose of this article is to bring attention to the problem of variations in relevance assessments and the effects that these may have on measures of retrieval effectiveness. Through an analytical review of the literature, I show that despite known wide variations in relevance assessments in experimental test collections, their effects on the measurement of retrieval performance are almost completely unstudied. I will further argue that what we know about the many variables that have been found to affect relevance assessments under experimental conditions, as well as our new understanding of psychological, situational, user‐based relevance, point to a single conclusion. We can no longer rest the evaluation of information retrieval systems on the assumption that such variations do not significantly affect the measurement of information retrieval performance. A series of thorough, rigorous, and extensive tests is needed, of precisely how, and under what conditions, variations in relevance assessments do, and do not, affect measures of retrieval performance. We need to develop approaches to evaluation that are sensitive to these variations and to human factors and individual differences more generally. Our approaches to evaluation must reflect the real world of real users. © 1996 John Wiley & Sons, Inc.