Cumulated gain-based evaluation of IR techniques
Cumulated gain-based evaluation of IR techniques
复制标题
DOI：
10.1145/582415.582418
复制
发表时间：
2002-10-01
影响因子：
5.6
通讯作者：
Kekäläinen, J
中科院分区：
文献类型：
--
作者：
Järvelin, K;Kekäläinen, J
现代大型检索环境往往因其庞大的输出而使用户不堪重负。由于并非所有文档与其用户都具有同等的相关性，因此应确定高度相关的文档并将其排在首位以供展示。为了在这个方向上发展IR技术，有必要开发评估方法和方法，使IR方法能够检索高度相关的文件。这可以通过将传统的评价方法，即基于二值关联判断的查全率和查准率扩展到分级关联判断来实现。或者，可以开发基于分级相关性判断的新措施。本文提出了几种新的度量方法来计算用户通过检查检索结果获得的累积增益，直到给定的排名位置。第一个方法根据排序的结果列表累积检索文档的相关性分数。第二种方法类似，但对相关性分数应用折扣因子，以便对晚检索的文档进行贬值。第三个是计算红外技术相对于理想的性能，基于它们能够产生的累积增益。定义和讨论了这些新度量，并在使用TREC数据的案例研究中演示了它们的使用：TREC-7中20个查询的示例系统运行结果。作为关联基础，我们使用了新的四级等级关联判断。测试结果表明，所提出的措施认为IR方法具有检索高度相关文件的能力，并允许测试有效性差异的统计显著性。基于度量的图表还提供了对性能IR技术的洞察，并允许从用户的角度进行解释。
Modern large retrieval environments tend to overwhelm their users by their large output. Since all documents are not of equal relevance to their users, highly relevant documents should be identified and ranked first for presentation. In order to develop IR techniques in this direction, it is necessary to develop evaluation approaches and methods that credit IR methods for their ability to retrieve highly relevant documents. This can be done by extending traditional evaluation methods, that is, recall and precision based on binary relevance judgments, to graded relevance judgments. Alternatively, novel measures based on graded relevance judgments may be developed. This article proposes several novel measures that compute the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. The first one accumulates the relevance scores of retrieved documents along the ranked result list. The second one is similar but applies a discount factor to the relevance scores in order to devaluate late-retrieved documents. The third one computes the relative-to-the-ideal performance of IR techniques, based on the cumulative gain they are able to yield. These novel measures are defined and discussed and their use is demonstrated in a case study using TREC data: sample system run results for 20 queries in TREC-7. As a relevance base we used novel graded relevance judgments on a four-point scale. The test results indicate that the proposed measures credit IR methods for their ability to retrieve highly relevant documents and allow testing of statistical significance of effectiveness differences. The graphs based on the measures also provide insight into the performance IR techniques and allow interpretation, for example, from the user point of view.