Improving Experimentation and Measurement for Online Products and Services

近年来，做出数据驱动或数据知情决策的价值变得越来越明显。做出数据驱动决策的关键是准确衡量给定选择的影响并试验可能的替代方案的能力。我们将实验和测量(E&M)能力定义为使用不同的数字或物理产品、服务或体验运行实验(受控或以其他方式进行)并衡量其影响所需的知识和工具。这些能力可能是以在线受控实验框架、计量经济学分析师团队或能够执行机器学习辅助因果推理的系统的形式-基于观察到的对因果关系的理解。研究项目旨在开发一系列统计和机器学习方法，以提高在线企业的E&M能力。到目前为止，我们已经成功地(1)估计了机电能力本身的价值，(2)建立了受控实验设计的评估框架，并寻求：(3)开发数据驱动的受控实验设计：在Web上运行许多大型受控实验的能力使我们能够收集过去类似实验产生的数据。如果实验的结果趋势相似，我们能否利用这些数据来缩短实验的持续时间？(4)先验地量化观察性研究的测量不确定性：观察性研究(没有控制组的实验)允许人们使用因果推理来估计行动/属性的影响的方向和程度。与受控的同行不同，围绕估计的不确定性水平通常只有在运行分析后才知道。我们能否像对对照实验那样，在运行这类分析之前量化估算结果的界限？(5)将对照实验和观察性实验产生的洞察力结合起来：与实验科学不同，在实验科学中，数据通常是为特定目的而收集的，而网络上的数据往往被用于最初打算之外的目的。我们能否对实验做同样的事情，即用为其他目的设计的对照实验产生的数据来补充观察性研究(反之亦然)？研究项目的范围是雄心勃勃的。由于需要在多个相关领域得出最新成果，许多挑战(专题1、2和5)由于需要在多个相关领域取得最新成果，在我们所知的范围内几乎没有得到考虑或认真尝试。其他问题需要建立在贝叶斯假设检验(主题3)和极值理论(主题2和4)的最新基础上。对于前者，我们计划将数据驱动的先验和非本地先验结合起来，以使实验对业务指标的任何变化更加敏感。数据驱动的先验是指基于以前的实验如何执行的规范，非本地先验是关于实验如何可能进行的规范，不违反零假设。对于后者，我们寻求降低面对重尾分布式响应时产生的估计的不确定性水平，即可以使用80/20规则来表征的极其广泛的在线行为。该项目属于EPSRC数字经济主题，涉及行为研究；数据、信息和知识；以及价值创造和获取等子主题。它是与英国最大的在线时尚零售商之一ASOS.com合作进行的，并得到了ASOS.com的部分资金支持。学术和行业的合作使研究能够访问丰富的适当数据，这些数据将为测量方法的发展提供信息，并使企业受益于立即开发的见解和技术，在迭代开发的过程中生成新的数据。

The value of making data driven or data informed decisions has become increasingly clear in recent years. Key to making data driven decisions is the ability to accurately measure the impact of a given choice and to experiment with possible alternatives. We define Experimentation & Measurement (E&M) capabilities as the knowledge and tools necessary to run experiments (controlled or otherwise) with different digital or physical products, services, or experiences, and measure their impact. The capabilities may be in the form of an online controlled experiment framework, a team of econometrics analysts, or a system capable of performing machine learning-aided causal inference---the understanding of the cause and effect based on what one observed.The research project aims to develop an array of statistical and machine learning methods, to boost the E&M capabilities of online businesses. So far, we have successfully (1) estimated the value of E&M capabilities themselves and (2) built an evaluation framework for controlled experiment designs, and seek to: (3) Develop data-driven controlled experiment designs:The ability to run many large-scale controlled experiments on the Web allows us to collect data generated by similar experiments in the past. Can we leverage these data to shorten the duration of an experiment if it yields results on a similar trend?(4) Quantify the measurement uncertainty of observational studies a priori:Observational studies (experiments w/o a control group) allow one to estimate both the direction and magnitude of the impact of an action/attribute using causal inference. Unlike its controlled counterpart, the uncertainty level around an estimate is often known only after running the analysis. Can we quantify the bounds on estimates produced from such analyses before running them, just like how we do so for controlled experiments?(5) Combine insights generated from controlled and observational experiments:Unlike in experimental science, where data are generally collected for a specific purpose, data on the Web are often used for purposes other than that originally intended. Can we do the same with experiments, i.e. supplement observational studies with data generated from controlled experiments that were designed for other purposes (or vice versa)?The scope of the research project is ambitious. Many of the challenges (topics 1, 2, and 5) had received little consideration or serious attempts to the best of our knowledge due to the need to draw the latest results in multiple related fields. Other problems require building on the state-of-the-art in Bayesian Hypothesis Testing (topic 3) and Extreme Value Theory (topics 2 and 4). For the former we plan to combine Data-driven Priors, the specification on how a new experiment could possibly perform based on how previous experiments performed, and Non-local Priors, the specification on how an experiment could possibly perform that does not contradict the null hypothesis, to make an experiment more sensitive to any changes in a business metric. For the latter we seek to reduce the uncertainty level of the estimates produced in the face of heavy-tailed distributed responses, i.e. an extremely wide range of online behaviour that can be characterised using the 80/20 rule.The project falls within the EPSRC Digital Economy Theme, touching on sub-themes including Behavioural research; Data, information and knowledge; and Value creation and capture. It is carried out in collaboration with, and part-funded by, ASOS.com, one of the largest UK-based online fashion retailers. The academic-industry collaboration enables the research to access the wealth of what would be propriety data that informs the development of measurement methods, and the business to benefit from insights and techniques developed immediately, generating new data along the way for iterative development.