Counterfactual Estimation and Optimization of Click Metrics for Search Engines
Counterfactual Estimation and Optimization of Click Metrics for Search Engines
复制标题
搜索引擎点击指标的反事实估计和优化
DOI：
--
复制
发表时间：
2014
期刊：
影响因子：
--
通讯作者：
Ankur Gupta
中科院分区：
文献类型：
--
作者：
Lihong Li;Shunbao Chen;Jim Kleban;Ankur Gupta
当通过用户反馈（如点击和付款）计算指标时，针对预定义的在线度量标准优化交互式系统特别具有挑战性。关键挑战是反事实性质：在Web搜索的情况下，搜索引擎组成部分的任何更改都可能导致相同查询的另一个搜索结果页面，但是我们通常无法从搜索日志可靠地推断出用户如何反应反应到新的结果页面。因此，除非运行新引擎以服务用户并与A/B测试中的基线进行比较，否则似乎不可能准确估算依赖用户反馈的在线指标。不幸的是，这种方法虽然有效且成功，但既昂贵又耗时。在本文中，我们建议使用因果推理技术在上下文伴奏框架下解决这个问题。这种方法有效地使人们可以（可能无限地）从搜索日志中运行许多A/B测试，从而可以快速，廉价地估算和优化在线指标。为了关注商业搜索引擎中的重要组成部分，我们展示了如何实例化和应用这些想法，并获得非常有希望的结果，这些结果表明这些技术的适用性广泛。
Optimizing an interactive system against a predefined online metric is particularly challenging, when the metric is computed from user feedback such as clicks and payments. The key challenge is the counterfactual nature: in the case of Web search, any change to a component of the search engine may result in a different search result page for the same query, but we normally cannot infer reliably from search log how users would react to the new result page. Consequently, it appears impossible to accurately estimate online metrics that depend on user feedback, unless the new engine is run to serve users and compared with a baseline in an A/B test. This approach, while valid and successful, is unfortunately expensive and time-consuming. In this paper, we propose to address this problem using causal inference techniques, under the contextual-bandit framework. This approach effectively allows one to run (potentially infinitely) many A/B tests offline from search log, making it possible to estimate and optimize online metrics quickly and inexpensively. Focusing on an important component in a commercial search engine, we show how these ideas can be instantiated and applied, and obtain very promising results that suggest the wide applicability of these techniques.