RI: Small: Collaborative Research: Batch Learning from Logged Bandit Feedback

日志数据是可用的最普遍的数据形式之一，因为它可以从各种系统(例如，搜索引擎、推荐系统、广告投放平台)以很低的成本被记录。使学习算法能够访问大量的日志数据提供了以前所未有的规模获取知识的潜力。此外，即使在人工标记训练数据在经济上不可行的系统中，从日志数据中学习的能力也可以实现有效的机器学习。然而，日志数据仅提供限于系统采取的特定行动的部分信息--“背景--强盗反馈”。系统可能采取的所有其他行动的反馈通常是未知的。这使得从日志数据中进行学习从根本上不同于传统的监督学习，在传统的监督学习中，“正确的”预测与损失函数一起提供全信息反馈。该项目通过开发原则性的学习方法和算法来解决从强盗反馈中批量学习(BLBF)的问题，这些方法和算法可以用包含上下文-强盗反馈的日志来训练。首先，该项目发展了BLBF的学习理论，特别是关于理解BLBF的反事实风险估计器的使用和设计。其次，该项目为BLBF衍生出了新的学习方法。过去的工作已经证明，条件随机场可以在BLBF设置中进行训练，该项目也推导出了其他学习方法的BLBF模拟。第三，该项目为这些BLBF方法推导出可扩展的训练算法，以支持大规模应用。最后，该项目使用来自操作系统的真实数据验证了这些方法。

Log data is one of the most ubiquitous forms of data available, as it can be recorded from a variety of systems (e.g., search engines, recommender systems, ad placement platforms) at little cost. Making huge amounts of log data accessible to learning algorithms provides the potential to acquire knowledge at unprecedented scale. Furthermore, the ability to learn from log data can enable effective machine learning even in systems where manual labeling of training data is not economically viable. Log data, however, provides only partial information -- "contextual-bandit feedback" -- limited to the particular actions taken by the system. The feedback for all the other actions the system could have taken is typically not known. This makes learning from log data fundamentally different from traditional supervised learning, where "correct" predictions together with a loss function provide full-information feedback.This project tackles the problem of Batch Learning from Bandit Feedback (BLBF) by developing principled learning methods and algorithms that can be trained with logs containing contextual-bandit feedback. First, the project develops the learning theory of BLBF, especially with respect to understanding the use and design of counterfactual risk estimators for BLBF. Second, the project derives new learning methods for BLBF. Past work has already demonstrated that Conditional Random Fields can be trained in the BLBF setting, and the project derives BLBF analogs of other learning methods as well. Third, the project derives scalable training algorithms for these BLBF methods to enable large-scale applications. And, finally, the project validates the methods with real-world data from operational systems.