Towards Reliable Item Sampling for Recommendation Evaluation
Towards Reliable Item Sampling for Recommendation Evaluation
复制标题
DOI：
10.48550/arxiv.2211.15743
复制
发表时间：
2022-11
期刊：
影响因子：
--
通讯作者：
Dong Li;Ruoming Jin;Zhenming Liu;Bin Ren;Jing Gao;Zhi Liu
中科院分区：
文献类型：
--
作者：
Dong Li;Ruoming Jin;Zhenming Liu;Bin Ren;Jing Gao;Zhi Liu
由于Rendle和Krichene认为常用的基于抽样的评估指标与全局指标“不一致”（即使在预期中），因此已有一些关于基于抽样的推荐系统评估的研究。现有的方法尝试将基于采样的度量映射到它们的全局对应度量，或者更一般地，学习经验秩分布来估计前K个度量。然而，尽管已有努力，但对所提出的度量估计量仍然缺乏严格的理论理解，基本项目抽样也存在“盲点”问题，即，当K很小时，恢复前K个度量的估计精度仍然相当大。在本文中，我们提供了一个深入的调查，这些问题，并作出两个创新的贡献。首先，我们提出了一个新的项目抽样估计，显式优化的误差相对于地面真理，并在理论上突出了其与以前的工作的细微差别。其次，我们提出了一种新的自适应采样方法，旨在处理的“盲点”的问题，并证明期望最大化（EM）算法可以推广到这样的设置。我们的实验结果证实了我们的统计分析和所提出的工作的优越性。该研究为采用项目抽样度量进行推荐评价奠定了理论基础，为项目抽样成为一种强有力的、可靠的推荐评价工具提供了有力的证据。
Since Rendle and Krichene argued that commonly used sampling-based evaluation metrics are ``inconsistent'' with respect to the global metrics (even in expectation), there have been a few studies on the sampling-based recommender system evaluation. Existing methods try either mapping the sampling-based metrics to their global counterparts or more generally, learning the empirical rank distribution to estimate the top-K metrics. However, despite existing efforts, there is still a lack of rigorous theoretical understanding of the proposed metric estimators, and the basic item sampling also suffers from the ``blind spot'' issue, i.e., estimation accuracy to recover the top-K metrics when K is small can still be rather substantial. In this paper, we provide an in-depth investigation into these problems and make two innovative contributions. First, we propose a new item-sampling estimator that explicitly optimizes the error with respect to the ground truth, and theoretically highlights its subtle difference against prior work. Second, we propose a new adaptive sampling method that aims to deal with the ``blind spot'' problem and also demonstrate the expectation-maximization (EM) algorithm can be generalized for such a setting. Our experimental results confirm our statistical analysis and the superiority of the proposed works. This study helps lay the theoretical foundation for adopting item sampling metrics for recommendation evaluation and provides strong evidence for making item sampling a powerful and reliable tool for recommendation evaluation.