Improving Experimentation and Measurement for Online Products and Services

点击翻译按钮获取中文摘要

The value of making data driven or data informed decisions has become increasingly clear in recent years. Key to making data driven decisions is the ability to accurately measure the impact of a given choice and to experiment with possible alternatives. We define Experimentation & Measurement (E&M) capabilities as the knowledge and tools necessary to run experiments (controlled or otherwise) with different digital or physical products, services, or experiences, and measure their impact. The capabilities may be in the form of an online controlled experiment framework, a team of econometrics analysts, or a system capable of performing machine learning-aided causal inference---the understanding of the cause and effect based on what one observed.The research project aims to develop an array of statistical and machine learning methods, to boost the E&M capabilities of online businesses. So far, we have successfully (1) estimated the value of E&M capabilities themselves and (2) built an evaluation framework for controlled experiment designs, and seek to: (3) Develop data-driven controlled experiment designs:The ability to run many large-scale controlled experiments on the Web allows us to collect data generated by similar experiments in the past. Can we leverage these data to shorten the duration of an experiment if it yields results on a similar trend?(4) Quantify the measurement uncertainty of observational studies a priori:Observational studies (experiments w/o a control group) allow one to estimate both the direction and magnitude of the impact of an action/attribute using causal inference. Unlike its controlled counterpart, the uncertainty level around an estimate is often known only after running the analysis. Can we quantify the bounds on estimates produced from such analyses before running them, just like how we do so for controlled experiments?(5) Combine insights generated from controlled and observational experiments:Unlike in experimental science, where data are generally collected for a specific purpose, data on the Web are often used for purposes other than that originally intended. Can we do the same with experiments, i.e. supplement observational studies with data generated from controlled experiments that were designed for other purposes (or vice versa)?The scope of the research project is ambitious. Many of the challenges (topics 1, 2, and 5) had received little consideration or serious attempts to the best of our knowledge due to the need to draw the latest results in multiple related fields. Other problems require building on the state-of-the-art in Bayesian Hypothesis Testing (topic 3) and Extreme Value Theory (topics 2 and 4). For the former we plan to combine Data-driven Priors, the specification on how a new experiment could possibly perform based on how previous experiments performed, and Non-local Priors, the specification on how an experiment could possibly perform that does not contradict the null hypothesis, to make an experiment more sensitive to any changes in a business metric. For the latter we seek to reduce the uncertainty level of the estimates produced in the face of heavy-tailed distributed responses, i.e. an extremely wide range of online behaviour that can be characterised using the 80/20 rule.The project falls within the EPSRC Digital Economy Theme, touching on sub-themes including Behavioural research; Data, information and knowledge; and Value creation and capture. It is carried out in collaboration with, and part-funded by, ASOS.com, one of the largest UK-based online fashion retailers. The academic-industry collaboration enables the research to access the wealth of what would be propriety data that informs the development of measurement methods, and the business to benefit from insights and techniques developed immediately, generating new data along the way for iterative development.