EAGER: Collaborative Research: On the Theoretical Foundation of Recommendation System Evaluation

该项目为评估推荐系统（RS）的性能开发了一个新的理论基础，推荐系统是指导在线用户和购物者在大量产品和网站中导航的关键组成部分。尽管2019冠状病毒病大流行，但2020年美国在线零售总额接近1万亿美元。由于网上购物预计会增加，适当的RS设计将改善购物/浏览，帮助小型网上企业生存，并为国家经济做出贡献。最近的研究已经注意到基于深度学习的推荐获得了相当大的改进。然而，一些研究表明，这些改进可能是虚假的，因为设计不良的实验、选择不当的基线、精心挑选的数据集、不准确的RS性能指标，以及使用无效的评估协议，导致评估和生产环境之间的性能差异。认识到基线和数据集问题可以通过使用标准基准来解决，该项目侧重于设计可靠的新计算工具、指标和评估协议来分析推荐系统。这些工具将包括基于用户行为的精确统计模型对RS进行评分的新方法，以及一套使用更少样本和计算资源的新算法，从而产生更准确的性能估计。从技术角度来看，该项目将开发基于统计学习理论和随机过程的理论工具来分析RS的评估指标和协议。该项目侧重于三个任务。首先，设计有效的度量估计程序，通过统一最近提出的两种基于采样和搜索总体最佳估计器来恢复top-K度量的临时方法，解决采样和top-K评估度量（例如，归一化贴现累积增益（nDCG）和召回率）之间的不匹配。其次，开发了量化top-K指标的敏感性和鲁棒性的方法，并设计了新的项目抽样程序，以提高现有指标的鲁棒性。最后，该项目将分析离线评估与生产环境（在线设置）之间的性能差距，并提出了一个新的离线评估指标，可以更好地模拟在线性能。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project develops a new theoretical foundation for evaluating the performance of recommendation systems (RS), a crucial component guiding online users and shoppers to navigate a sea of products and websites. Despite the Covid-19 pandemic, online retail sales in the US totaled nearly $1 trillion dollars in 2020. Since online purchasing is forecasted to increase, proper design of RS will improve shopping/browsing, help small online businesses to survive, and contribute to the nation’s economy. Recent studies have noted the sizeable improvements obtained from deep learning-based recommendations. However, several studies suggest that these improvements may be spurious due to poorly designed experiments with ill-chosen baselines, cherry-picked datasets, inaccurate metrics of RS performance, and the use of ineffective evaluation protocols that result in performance discrepancies between evaluation and production environments. Recognizing that baseline and dataset problems can be addressed by using standard benchmarks, this project focuses on designing reliable new computation tools, metrics, and evaluation protocols for analyzing recommendation systems. The tools will include new ways to score an RS based on accurate statistical models of user behaviors and a suite of new algorithms that use fewer samples and computational resources that produce more accurate estimations of performance.From a technical standpoint, this project will develop theoretical tools to analyze evaluation metrics and protocols for RS based on statistical learning theory and stochastic processes. The project focuses on three tasks. First, designing efficient metrics estimation procedures that resolve the mismatch between sampling and top-K evaluation metrics (e.g., normalized discounted cumulative gain (nDCG) and Recall) by unifying two recently proposed ad hoc approaches for recovering the top-K metrics based on sampling and searching for an overall best estimator. Second, the develops methods to quantify the sensitivity and robustness of the top-K metrics, and design new item sampling procedures that improve the robustness of existing metrics, The finally, the project will analyze the performance gap between offline evaluations and production environments (the online settings), and proposing a new offline evaluation metrics that can better mimic online performance.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.