Adaptive Design for Fast Machine/Statistical Learning

点击翻译按钮获取中文摘要

The overarching goal of the research program is to extend Gaussian processes (GPs) to enable much more complex applications. First, new methodology will scale up GPs to enable large sample sizes and use adaptive sampling, for accurate statistical modelling of complex relationships arising cross a broad spectrum of scientific and engineering disciplines. Second, efficiencies in adaptive search methods using GPs will allow automatic tuning of computationally intensive machine/statistical learners. GPs have had profound impact on science and engineering, where they are used directly as machine/statistical learners. Complex computer codes of physical systems can be too slow for optimization, calibration of unknowns, sensitivity analysis, etc.  GPs trained on limited computer model runs are used for these purposes as computationally fast surrogates for the particular scientific objective. It is well known, however, that the computational time to train a GP increases as the cube of the sample size. Thus, GPs are less attractive for sample sizes of a few thousand or more.  Existing methods, mainly based on localized modelling or special fixed experimental designs, will be assessed to determine the domain of problems where they are effective.  It is clear in advance, however, that new methods will be required for complex applications: those with moderate to high-dimensional input, nonlinear relationships, and/or high-order interaction effects. Only by adapting the experiment - taking further observations where the target function has special features - can a dense sampling of the input space be obtained where it matters. Divide and conquer methods are especially promising. How to divide high-dimensional space, how to choose sub-regions for data augmentation, and guidance on the number of new runs per iteration will be critical research questions here. GPs are also used indirectly in support of other machine-learning (ML) methods such as deep learning neural networks. Neural networks for image classification, for example, have "tuning" parameters that have to be set by the user, to determine the basic network architecture or regularization, for instance.  Users tune these so-called hyperparameters by trying different values and attempting to minimize validation error in various ways. To obtain the validation error requires training the ML method, which is itself computationally very intensive. Hence, systematic methods known as Bayesian optimization train a GP to model the relationship between the hyperparameter settings and validation error, and hence adaptively optimize the error. The research program will continue work in my lab on "automatic ML", to minimize the number of tries of the expensive underlying ML algorithm. Advances here will likely have impact on other computationally challenging optimization problems where the objective is produced by an expensive algorithm.