Improving the Sensitivity of Online Controlled Experiments: Case Studies at Netflix
Improving the Sensitivity of Online Controlled Experiments: Case Studies at Netflix
复制标题
提高在线控制实验的灵敏度：Netflix 的案例研究
DOI：
10.1145/2939672.2939733
复制
发表时间：
2016
期刊：
影响因子：
--
通讯作者：
Juliette Aurisset
中科院分区：
文献类型：
--
作者：
Huizhi Xie;Juliette Aurisset
控制实验被广泛认为是建立产品变化及其对业务指标影响之间真正因果关系的最科学的方法。许多科技公司依赖此类实验作为其主要的数据驱动决策工具。受控实验的灵敏度是指其检测由于产品变化而导致的业务指标差异的能力。对于拥有数千万用户的Netflix来说，提高受控实验的灵敏度至关重要，因为未能检测到微小的影响，无论是积极的还是消极的，都可能对收入产生重大影响。本文重点研究了通过减小业务指标的抽样方差来提高灵敏度的方法。我们定义了Netflix的业务指标，并围绕提高敏感度的关键需求分享了相关内容。我们回顾了广泛适用于任何类型的控制实验和度量的流行方差缩减技术。我们描述了Netflix分层抽样的一种创新实现，用户被实时分配到实验中，并讨论了实现过程中一些令人惊讶的挑战。我们进行了案例研究，在一些Netflix数据集上比较这些方差减少技术。基于实证结果，我们建议在大规模对照实验中使用后分层和CUPED等分配后方差减少技术来代替分层抽样等分配时方差减少技术。
Controlled experiments are widely regarded as the most scientific way to establish a true causal relationship between product changes and their impact on business metrics. Many technology companies rely on such experiments as their main data-driven decision-making tool. The sensitivity of a controlled experiment refers to its ability to detect differences in business metrics due to product changes. At Netflix, with tens of millions of users, increasing the sensitivity of controlled experiments is critical as failure to detect a small effect, either positive or negative, can have a substantial revenue impact. This paper focuses on methods to increase sensitivity by reducing the sampling variance of business metrics. We define Netflix business metrics and share context around the critical need for improved sensitivity. We review popular variance reduction techniques that are broadly applicable to any type of controlled experiment and metric. We describe an innovative implementation of stratified sampling at Netflix where users are assigned to experiments in real time and discuss some surprising challenges with the implementation. We conduct case studies to compare these variance reduction techniques on a few Netflix datasets. Based on the empirical results, we recommend to use post-assignment variance reduction techniques such as post stratification and CUPED instead of at-assignment variance reduction techniques such as stratified sampling in large-scale controlled experiments.