权益分类	功能权益	普通用户	{{item.name}}会员
{{category.name}}	{{benefitItem.name}}

Improving Experimentation and Measurement for Online Products and Services

改进在线产品和服务的实验和测量

基本信息

批准号：
2284224
负责人：
金额：
--
依托单位：
Imperial College London
依托单位国家：
英国
项目类别：
Studentship
财政年份：
2019
资助国家：
英国
起止时间：
2019 至无数据
项目状态：
已结题

来源：
https://gtr.ukri.org/projects?ref=studentship-2284224
关键词：
Improving Experimentation Measurement Online Products

项目摘要

The value of making data driven or data informed decisions has become increasingly clear in recent years. Key to making data driven decisions is the ability to accurately measure the impact of a given choice and to experiment with possible alternatives. We define Experimentation & Measurement (E&M) capabilities as the knowledge and tools necessary to run experiments (controlled or otherwise) with different digital or physical products, services, or experiences, and measure their impact. The capabilities may be in the form of an online controlled experiment framework, a team of econometrics analysts, or a system capable of performing machine learning-aided causal inference---the understanding of the cause and effect based on what one observed.The research project aims to develop an array of statistical and machine learning methods, to boost the E&M capabilities of online businesses. So far, we have successfully (1) estimated the value of E&M capabilities themselves and (2) built an evaluation framework for controlled experiment designs, and seek to: (3) Develop data-driven controlled experiment designs:The ability to run many large-scale controlled experiments on the Web allows us to collect data generated by similar experiments in the past. Can we leverage these data to shorten the duration of an experiment if it yields results on a similar trend?(4) Quantify the measurement uncertainty of observational studies a priori:Observational studies (experiments w/o a control group) allow one to estimate both the direction and magnitude of the impact of an action/attribute using causal inference. Unlike its controlled counterpart, the uncertainty level around an estimate is often known only after running the analysis. Can we quantify the bounds on estimates produced from such analyses before running them, just like how we do so for controlled experiments?(5) Combine insights generated from controlled and observational experiments:Unlike in experimental science, where data are generally collected for a specific purpose, data on the Web are often used for purposes other than that originally intended. Can we do the same with experiments, i.e. supplement observational studies with data generated from controlled experiments that were designed for other purposes (or vice versa)?The scope of the research project is ambitious. Many of the challenges (topics 1, 2, and 5) had received little consideration or serious attempts to the best of our knowledge due to the need to draw the latest results in multiple related fields. Other problems require building on the state-of-the-art in Bayesian Hypothesis Testing (topic 3) and Extreme Value Theory (topics 2 and 4). For the former we plan to combine Data-driven Priors, the specification on how a new experiment could possibly perform based on how previous experiments performed, and Non-local Priors, the specification on how an experiment could possibly perform that does not contradict the null hypothesis, to make an experiment more sensitive to any changes in a business metric. For the latter we seek to reduce the uncertainty level of the estimates produced in the face of heavy-tailed distributed responses, i.e. an extremely wide range of online behaviour that can be characterised using the 80/20 rule.The project falls within the EPSRC Digital Economy Theme, touching on sub-themes including Behavioural research; Data, information and knowledge; and Value creation and capture. It is carried out in collaboration with, and part-funded by, ASOS.com, one of the largest UK-based online fashion retailers. The academic-industry collaboration enables the research to access the wealth of what would be propriety data that informs the development of measurement methods, and the business to benefit from insights and techniques developed immediately, generating new data along the way for iterative development.

近年来，数据驱动或数据知情决策的价值越来越明显。做出数据驱动决策的关键是能够准确衡量给定选择的影响，并尝试可能的替代方案。我们将实验与测量（E&M）能力定义为对不同的数字或物理产品、服务或体验进行实验（受控或其他）并测量其影响所需的知识和工具。这些能力可以是在线控制实验框架、计量经济学分析师团队或能够执行机器学习辅助因果推理的系统的形式。该研究项目旨在开发一系列统计和机器学习方法，以提高在线业务的E&M能力。到目前为止，我们已经成功地（1）估计了E&M能力本身的价值，（2）建立了对照实验设计的评估框架，并寻求：（3）开发数据驱动的对照实验设计：在Web上运行许多大规模对照实验的能力使我们能够收集过去类似实验产生的数据。如果一个实验产生了类似趋势的结果，我们能否利用这些数据来缩短实验的持续时间？(4)先验量化观察性研究的测量不确定性：观察性研究（实验w/o控制组）允许使用因果推理来估计行动/属性的影响的方向和幅度。与其受控对应物不同，估计值的不确定性水平通常只有在运行分析后才知道。我们能否在运行这些分析之前量化这些分析所产生的估计值的界限，就像我们在控制实验中所做的那样？(5)联合收割机从控制和观察实验中产生的见解：与实验科学不同，在实验科学中，数据通常是为特定目的收集的，而Web上的数据通常用于最初意图之外的目的。我们能否对实验做同样的事情，即用为其他目的设计的对照实验产生的数据补充观察性研究（反之亦然）？该研究项目的范围是雄心勃勃的。据我们所知，由于需要在多个相关领域取得最新成果，许多挑战（主题1、2和5）几乎没有得到考虑或认真尝试。其他问题需要建立在贝叶斯假设检验（主题3）和极值理论（主题2和4）的最新技术基础上。对于前者，我们计划结合联合收割机数据驱动先验（Data-driven Priors）和非局部先验（Non-local Priors），前者是关于新实验可能如何执行的规范，基于先前实验的执行方式，后者是关于实验可能如何执行的规范，不与零假设相矛盾，以使实验对业务指标的任何变化更敏感。对于后者，我们寻求降低在面对厚尾分布响应时产生的估计的不确定性水平，即可以使用80/20规则描述的极其广泛的在线行为。该项目福尔斯EPSRC数字经济主题，涉及行为研究;数据，信息和知识;以及价值创造和捕获等子主题。它是与英国最大的在线时尚零售商之一ASOS.com合作开展的，并由其提供部分资金。学术界与行业的合作使研究人员能够获得丰富的适当数据，这些数据为测量方法的开发提供了信息，而业务部门则可以从立即开发的见解和技术中受益，从而沿着迭代开发的道路生成新数据。