CAREER: User-Based Simulation Methods for Quantifying Sources of Error and Bias in Recommender Systems

推荐产品、地点和服务的系统在日常生活和商业中越来越普遍，因此了解推荐算法如何影响个人用户和更大的社会群体的结果非常重要。为此，项目团队将开发基于大规模历史数据集模拟用户行为的新方法。这些方法将用于更好地理解训练数据集中潜在的偏见对常用的基于机器学习的构建和测试推荐系统的方法构成的漏洞，以及表征常见评估指标（如推荐准确性和多样性）的有效性，给出人们在实践中如何与推荐系统交互的不同模型。该团队将公开发布其数据集、软件和新指标，以供其他推荐系统的研究人员和开发人员使用。这项工作还将为有关数据分析的社会影响的计算机科学课程材料的开发以及图书馆员的拓展活动提供信息，图书馆员经常帮助信息寻求者了解搜索引擎和其他推荐系统如何影响他们获得所需信息的能力。这部作品围绕两个主题展开。第一个将量化和减轻离线推荐评估中的流行偏差和错误分类诱饵问题，这些问题往往会导致流行的、已知的推荐。为了做到这一点，团队将开发基于模拟的评估模型，对用户如何选择相关商品进行购买和评级的各种假设进行编码，并使用它们来量化这些假设在推荐质量指标中引起的统计偏差。他们将通过与涵盖书籍、研究论文、音乐和电影的现有数据集进行比较来校准这些模拟。这些模型和数据集将有助于推动第二个主要项目，即测量训练数据中的特征分布对推荐算法准确性和多样性的影响，同时开发抗偏差算法。该团队将使用数据重采样技术以及模拟模型，扩展到随着时间的推移对系统行为进行建模，以评估不同的算法如何通过它们的推荐来减轻、传播或加剧潜在的分布偏差，以及这些有偏差的推荐如何反过来影响未来的用户行为和体验。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Systems that recommend products, places, and services are an increasingly common part of everyday life and commerce, making it important to understand how recommendation algorithms affect outcomes for both individual users and larger social groups. To do this, the project team will develop novel methods of simulating users' behavior based on large-scale historical datasets. These methods will be used to better understand vulnerabilities that underlying biases in training datasets pose to commonly-used machine learning-based methods for building and testing recommender systems, as well as characterize the effectiveness of common evaluation metrics such as recommendation accuracy and diversity given different models of how people interact with recommender systems in practice. The team will publicly release its datasets, software, and novel metrics for the benefit of other researchers and developers of recommender systems. The work also will inform the development of computer science course materials about the social impact of data analytics as well as outreach activities for librarians, who are often in the position of helping information seekers understand the way search engines and other recommender systems affect their ability to get what they need.The work is organized around two main themes. The first will quantify and mitigate the popularity bias and misclassified decoy problems in offline recommender evaluation that tend to lead to popular, known recommendations. To do this, the team will develop simulation-based evaluation models that encode a variety of assumptions about how users select relevant items to buy and rate and use them to quantify the statistical biases these assumptions induce in recommendation quality metrics. They will calibrate these simulations by comparing with existing data sets covering books, research papers, music, and movies. These models and datasets will help drive the second main project around measuring the impact of feature distributions in training data on recommender algorithm accuracy and diversity, while developing bias-resistant algorithms. The team will use data resampling techniques along with the simulation models, extended to model system behavior over time, to evaluate how different algorithms mitigate, propagate, or exacerbate underlying distributional biases through their recommendations, and how those biased recommendations in turn affect future user behavior and experience.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.