Directly optimizing evaluation measures in learning to rank
Directly optimizing evaluation measures in learning to rank
复制标题
DOI：
10.1145/1390334.1390355
复制
发表时间：
2008-07
期刊：
影响因子：
--
通讯作者：
Jun Xu;Tie-Yan Liu;Min Lu;Hang Li;Wei-Ying Ma
中科院分区：
文献类型：
--
作者：
Jun Xu;Tie-Yan Liu;Min Lu;Hang Li;Wei-Ying Ma
学习信息检索排名的核心问题之一是开发算法，通过直接优化信息检索中使用的评价指标（如平均平均精度（MAP）和归一化贴现累积增益（NDCG））来构建排名模型。已经提出了几种这样的算法，包括SVMmap和AdaRank，并验证了它们的有效性。然而，这些算法之间的关系并不清楚，而且没有在它们之间进行比较。研究了信息检索学习排序中直接优化评价指标的方法。我们专注于最小化损失函数上界的基本损失函数定义的IR措施的方法。我们首先提供了一个研究的一般框架，并在框架内分析了现有的SVM map和AdaRank算法。该框架是基于上限分析和两种类型的上限进行了讨论。此外，我们表明，我们可以推导出新的算法的基础上，这种分析，并创建一个示例算法称为PermuRank。我们还使用基准数据集对SVMmap、AdaRank、PermuRank以及传统的Ranking SVM和RankBoost方法进行了比较。实验结果表明，基于直接优化评价指标的方法总是优于传统的Ranking SVM和RankBoost方法。然而，直接优化方法本身的性能之间不存在显着差异。
One of the central issues in learning to rank for information retrieval is to develop algorithms that construct ranking models by directly optimizing evaluation measures used in information retrieval such as Mean Average Precision (MAP) and Normalized Discounted Cumulative Gain (NDCG). Several such algorithms including SVMmap and AdaRank have been proposed and their effectiveness has been verified. However, the relationships between the algorithms are not clear, and furthermore no comparisons have been conducted between them. In this paper, we conduct a study on the approach of directly optimizing evaluation measures in learning to rank for Information Retrieval (IR). We focus on the methods that minimize loss functions upper bounding the basic loss function defined on the IR measures. We first provide a general framework for the study and analyze the existing algorithms of SVMmap and AdaRank within the framework. The framework is based on upper bound analysis and two types of upper bounds are discussed. Moreover, we show that we can derive new algorithms on the basis of this analysis and create one example algorithm called PermuRank. We have also conducted comparisons between SVMmap, AdaRank, PermuRank, and conventional methods of Ranking SVM and RankBoost, using benchmark datasets. Experimental results show that the methods based on direct optimization of evaluation measures can always outperform conventional methods of Ranking SVM and RankBoost. However, no significant difference exists among the performances of the direct optimization methods themselves.