Adaptive Design for Fast Machine/Statistical Learning

该研究计划的首要目标是扩展高斯过程(GP)，以实现更复杂的应用。首先，新的方法将扩大全科医生的规模，以实现大样本量，并使用自适应抽样，对广泛的科学和工程学科产生的复杂关系进行准确的统计建模。其次，使用GP的自适应搜索方法的效率将允许对计算密集型机器/统计学习者进行自动调整。GPS对科学和工程产生了深远的影响，在这些领域，它们直接被用作机器/统计学习者。物理系统的复杂计算机代码对于优化、未知数校准、灵敏度分析等来说可能太慢。在有限的计算机模型运行中训练的GP用于这些目的，作为特定科学目标的计算快速替代品。然而，众所周知，训练GP的计算时间随着样本大小的立方体而增加。因此，对于几千或更多的样本量，GP就不那么有吸引力了。现有的方法，主要基于局部建模或特殊的固定实验设计，将进行评估，以确定它们有效的问题域。然而，预先很清楚，复杂的应用将需要新的方法：那些具有中到高维输入、非线性关系和/或高阶相互作用效应的方法。只有通过调整实验--在目标函数具有特殊特征的地方进行进一步的观察--才能在重要的地方获得输入空间的密集采样。分而治之的方法特别有前途。如何划分高维空间，如何选择用于数据增强的子区域，以及指导每次迭代的新运行数量将是这里的关键研究问题。GPS还间接用于支持其他机器学习(ML)方法，如深度学习神经网络。例如，用于图像分类的神经网络具有必须由用户设置的“调整”参数，以确定例如基本的网络结构或正则化。用户通过尝试不同的值并试图以各种方式最小化验证误差来调整这些所谓的超参数。为了获得验证误差，需要训练ML方法，这本身就是计算非常密集的。因此，被称为贝叶斯优化的系统方法训练GP来对超参数设置和验证误差之间的关系进行建模，从而自适应地优化误差。该研究计划将继续在我的实验室中进行“自动ML”的工作，以最大限度地减少昂贵的底层ML算法的尝试次数。这方面的进展可能会对其他具有计算挑战性的优化问题产生影响，在这些问题中，目标是由昂贵的算法产生的。

The overarching goal of the research program is to extend Gaussian processes (GPs) to enable much more complex applications. First, new methodology will scale up GPs to enable large sample sizes and use adaptive sampling, for accurate statistical modelling of complex relationships arising cross a broad spectrum of scientific and engineering disciplines. Second, efficiencies in adaptive search methods using GPs will allow automatic tuning of computationally intensive machine/statistical learners. GPs have had profound impact on science and engineering, where they are used directly as machine/statistical learners. Complex computer codes of physical systems can be too slow for optimization, calibration of unknowns, sensitivity analysis, etc.  GPs trained on limited computer model runs are used for these purposes as computationally fast surrogates for the particular scientific objective. It is well known, however, that the computational time to train a GP increases as the cube of the sample size. Thus, GPs are less attractive for sample sizes of a few thousand or more.  Existing methods, mainly based on localized modelling or special fixed experimental designs, will be assessed to determine the domain of problems where they are effective.  It is clear in advance, however, that new methods will be required for complex applications: those with moderate to high-dimensional input, nonlinear relationships, and/or high-order interaction effects. Only by adapting the experiment - taking further observations where the target function has special features - can a dense sampling of the input space be obtained where it matters. Divide and conquer methods are especially promising. How to divide high-dimensional space, how to choose sub-regions for data augmentation, and guidance on the number of new runs per iteration will be critical research questions here. GPs are also used indirectly in support of other machine-learning (ML) methods such as deep learning neural networks. Neural networks for image classification, for example, have "tuning" parameters that have to be set by the user, to determine the basic network architecture or regularization, for instance.  Users tune these so-called hyperparameters by trying different values and attempting to minimize validation error in various ways. To obtain the validation error requires training the ML method, which is itself computationally very intensive. Hence, systematic methods known as Bayesian optimization train a GP to model the relationship between the hyperparameter settings and validation error, and hence adaptively optimize the error. The research program will continue work in my lab on "automatic ML", to minimize the number of tries of the expensive underlying ML algorithm. Advances here will likely have impact on other computationally challenging optimization problems where the objective is produced by an expensive algorithm.