Limiting Bias from Test-Control Interference in Online Marketplace Experiments
Limiting Bias from Test-Control Interference in Online Marketplace Experiments
复制标题
限制在线市场实验中测试控制干扰的偏差
DOI：
--
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
Sinan Aral
中科院分区：
文献类型：
--
作者：
David Holtz;Sinan Aral
在A/B测试中，典型的目标是测量总平均治疗效果（TATE），其测量如果所有用户都被治疗的平均结果与如果所有用户都未被治疗的平均结果之间的差异。然而，当对照单位的结果取决于治疗单位的结果时，一个简单的均值差估计量将给出TATE的有偏估计，我们称之为测试-对照干扰的问题。使用建立在Airbnb数据之上的模拟，本文考虑使用网络干扰文献中的方法进行在线市场实验。我们将市场建模为一个网络，在这个网络中，如果两个卖家的商品可以相互替代，那么他们之间就存在一个优势。然后，我们模拟卖方的结果，特别是考虑“现状”的情况下，“治疗”的情况下，迫使所有卖家降低价格。我们使用相同的模拟框架来近似TATE分布，这些分布是通过使用块图聚类随机化、暴露建模和平均值差异的Hajek估计量产生的。我们发现，虽然块图聚类随机化降低了高达62%的天真的均值差异估计的偏差，它也显着增加了估计的方差。另一方面，使用更复杂的估计产生混合的结果。虽然有些提供了（小）额外的偏差减少和方差的小减少，其他导致增加的偏差和方差。总的来说，我们的研究结果表明，实验设计和分析技术，从网络实验文献是有前途的工具，减少偏见，由于测试控制干扰市场实验。
In an A/B test, the typical objective is to measure the total average treatment effect (TATE), which measures the difference between the average outcome if all users were treated and the average outcome if all users were untreated. However, a simple difference-in-means estimator will give a biased estimate of the TATE when outcomes of control units depend on the outcomes of treatment units, an issue we refer to as test-control interference. Using a simulation built on top of data from Airbnb, this paper considers the use of methods from the network interference literature for online marketplace experimentation. We model the marketplace as a network in which an edge exists between two sellers if their goods substitute for one another. We then simulate seller outcomes, specifically considering a "status quo" context and "treatment" context that forces all sellers to lower their prices. We use the same simulation framework to approximate TATE distributions produced by using blocked graph cluster randomization, exposure modeling, and the Hajek estimator for the difference in means. We find that while blocked graph cluster randomization reduces the bias of the naive difference-in-means estimator by as much as 62%, it also significantly increases the variance of the estimator. On the other hand, the use of more sophisticated estimators produces mixed results. While some provide (small) additional reductions in bias and small reductions in variance, others lead to increased bias and variance. Overall, our results suggest that experiment design and analysis techniques from the network experimentation literature are promising tools for reducing bias due to test-control interference in marketplace experiments.