Methods for Dealing with Misspecification in Bayesian Experimental Design

点击翻译按钮获取中文摘要

Much of the research in statistics and machine learning has focussed on methods of analysing data once it has already been acquired, but the question of how to best collect data in the first place has been under-explored. Gathering data can be expensive, therefore practitioners are limited by the amount of data they can collect. When this is done without care, it can produce poor quality data - potentially leading to inaccurate results and incorrect conclusions, regardless of how advanced one's analytical toolkit is. It is therefore vital to endeavour to gather good quality data before analysis. Research in experimental design aims to address this issue, providing practitioners with methods of collecting informative data that will lead to reliable results and strong conclusions. To outline Bayesian experimental design (BED), we consider the following setting: there are several beacons within an area, each emitting a signal, and a practitioner wishes to locate the beacons. The data-gathering process involves the practitioner choosing locations in which to probe the signal, then recording the strength of that signal at these locations. With infinite resources, the practitioner would be able to perfectly locate the beacons, but in practice they are constrained to performing only a finite number of experiments. BED then works as follows: before collecting any data, the practitioner will first form a statistical model of the strength of the signal at a given location in terms of the unknown locations of the beacons, and they will specify their prior beliefs about the locations of the beacons with a prior distribution on these locations. BED procedures can then provide the practitioner with the best locations to probe the signal - where "best" is defined as the locations that will lead to the largest increase in information about the beacon locations. The above can be easily generalised to other settings. BED is both theoretically sound and performs well practically, however, it can break down if our statistical model of the data is misspecified, i.e., if the true data-generating process is different from the model that we specified. Bayesian statistical methodology is always vulnerable to model misspecification, but unfortunately this fact is particularly problematic for experimental design, where we are not just using our model to analyse the data, but also to collect it in the first place. In the worst case, there are models in which the optimal course of action is to pick all your designs in exactly the same place, regardless of the outcomes you observe. However, unless your model is correct, this will produce an extremely poor quality dataset.In collaboration with my supervisor, Dr Tom Rainforth, we will first aim to deepen understanding of this problem: categorising the ways in which misspecification causes BED to fail; forming metrics to diagnose this failure and best practices to avoid its occurrence; and providing theoretical guarantees for when failure will occur. Following this, we will develop methods to counteract misspecification, ideally expanding the theoretical elegance and empirical performance of BED to cases when our model is misspecified. BED has enormous potential application, including quantum information experiments, psychology trials, constructing lifelike police sketches, and guiding drug discovery. As these applications become more complex, model misspecification becomes more prevalent; it is therefore pertinent to further investigate misspecification in BED. This project falls within the EPSRC 'statistics and applied probability' research area.