Beyond Relevance Ranking: A General Graph Matching Framework for Utility-Oriented Learning to Rank
Beyond Relevance Ranking: A General Graph Matching Framework for Utility-Oriented Learning to Rank
复制标题
DOI：
10.1145/3464303
复制
发表时间：
2021-11
期刊：
影响因子：
--
通讯作者：
Xinyi Dai;Yunjia Xi;Weinan Zhang;Qing Liu;Ruiming Tang;Xiuqiang He;Jiawei Hou;Jun Wang;Yong Yu
中科院分区：
文献类型：
--
作者：
Xinyi Dai;Yunjia Xi;Weinan Zhang;Qing Liu;Ruiming Tang;Xiuqiang He;Jiawei Hou;Jun Wang;Yong Yu
学习从记录的用户反馈（如点击或购买）中进行排名，是许多现实世界信息系统的核心组成部分。与人工标注的相关性标签不同，用户的反馈总是嘈杂和有偏见的。许多现有的学习排名方法推断的基础上的相关性的查询项目对不同的假设检查，并仍然优化基于相关性的目标。这种方法很大程度上依赖于对考试的正确估计，而这在实践中往往很难实现。在这项工作中，我们提出了一个通用的框架U-排名+学习排名与记录的用户反馈的角度图匹配。本文系统地分析了用户反馈中的偏差，包括检查偏差和选择偏差。然后，我们考虑到这两个偏见的无偏效用估计，直接基于用户反馈，而不是相关性。为了以有效的方式最大化估计的效用，我们设计了两个不同的求解器，基于Sinkhorn和LambdaLoss的U-秩+。前者是基于一个标准的图匹配算法，后者的灵感来自于传统的学习排名的方法。这两种算法在优化无偏效用目标方面都具有良好的理论性能，而后者在实践中被证明是更有效的。我们的框架U-rank+可以处理一个通用的效用函数，可以用于广泛的应用，包括网络搜索，推荐和在线广告。在三个基准学习数据集上的半合成实验证明了U-rank+的有效性。此外，我们提出的框架已经部署在两个不同的场景中的主流应用商店，其中在线A/B测试显示，U-rank+在推荐场景中实现了平均19.2%的点击率和20.8%的转化率提升，在线广告场景中的平台收入比生产基线提高了5.12%。
Learning to rank from logged user feedback, such as clicks or purchases, is a central component of many real-world information systems. Different from human-annotated relevance labels, the user feedback is always noisy and biased. Many existing learning to rank methods infer the underlying relevance of query–item pairs based on different assumptions of examination, and still optimize a relevance based objective. Such methods rely heavily on the correct estimation of examination, which is often difficult to achieve in practice. In this work, we propose a general framework U-rank+ for learning to rank with logged user feedback from the perspective of graph matching. We systematically analyze the biases in user feedback, including examination bias and selection bias. Then, we take both biases into consideration for unbiased utility estimation that directly based on user feedback, instead of relevance. In order to maximize the estimated utility in an efficient manner, we design two different solvers based on Sinkhorn and LambdaLoss for U-rank+. The former is based on a standard graph matching algorithm, and the latter is inspired by the traditional method of learning to rank. Both of the algorithms have good theoretical properties to optimize the unbiased utility objective while the latter is proved to be empirically more effective and efficient in practice. Our framework U-rank+ can deal with a general utility function and can be used in a widespread of applications including web search, recommendation, and online advertising. Semi-synthetic experiments on three benchmark learning to rank datasets demonstrate the effectiveness of U-rank+. Furthermore, our proposed framework has been deployed on two different scenarios of a mainstream App store, where the online A/B testing shows that U-rank+ achieves an average improvement of 19.2% on click-through rate and 20.8% improvement on conversion rate in recommendation scenario, and 5.12% on platform revenue in online advertising scenario over the production baselines.