A/B Testing with Fat Tails
A/B Testing with Fat Tails
复制标题
DOI：
10.1086/710607
复制
发表时间：
2020-12-01
影响因子：
8.2
通讯作者：
Weyl, E. Glen
中科院分区：
文献类型：
--
作者：
Azevedo, Eduardo M.;Deng, Alex;Weyl, E. Glen
我们提出了一个新的优化实验框架，我们称之为“A/B测试问题”。我们的模型与现有文献不同，允许厚尾。我们的关键见解是，最优战略取决于最大的收益是来自典型的创新，还是来自罕见的、不可预测的巨大成功。如果未观察到的创新质量分布的尾部不是太胖，那么使用几个高功率的“大”实验的标准方法是最优的。然而，如果分布是厚尾的，那么尝试更多想法的“精益”策略是首选的，每个想法的样本量可能更小。我们的理论结果以及对微软必应EXP平台的实证分析表明，对业务实践进行简单的改变就可以提高创新生产率。
We propose a new framework for optimal experimentation, which we term the "A/B testing problem." Our model departs from the existing literature by allowing for fat tails. Our key insight is that the optimal strategy depends on whether most gains accrue from typical innovations or from rare, unpredictable large successes. If the tails of the unobserved distribution of innovation quality are not too fat, the standard approach of using a few high-powered "big" experiments is optimal. However, if the distribution is very fat tailed, a "lean" strategy of trying more ideas, each with possibly smaller sample sizes, is preferred. Our theoretical results, along with an empirical analysis of Microsoft Bing's EXP platform, suggest that simple changes to business practices could increase innovation productivity.