Adaptive Design for Fast Machine/Statistical Learning

快速机器/统计学习的自适应设计

• 批准号: RGPIN-2019-05019
• 负责人: Welch, William
• 金额: $ 1.82万
• 依托单位: University of British Columbia
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2021
• 资助国家: 加拿大
• 起止时间: 2021-01-01 至 2022-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=738015
• 关键词: Adaptive; Design; Fast; Machine; Statistical

The overarching goal of the research program is to extend Gaussian processes (GPs) to enable much more complex applications. First, new methodology will scale up GPs to enable large sample sizes and use adaptive sampling, for accurate statistical modelling of complex relationships arising cross a broad spectrum of scientific and engineering disciplines. Second, efficiencies in adaptive search methods using GPs will allow automatic tuning of computationally intensive machine/statistical learners. GPs have had profound impact on science and engineering, where they are used directly as machine/statistical learners. Complex computer codes of physical systems can be too slow for optimization, calibration of unknowns, sensitivity analysis, etc.  GPs trained on limited computer model runs are used for these purposes as computationally fast surrogates for the particular scientific objective. It is well known, however, that the computational time to train a GP increases as the cube of the sample size. Thus, GPs are less attractive for sample sizes of a few thousand or more.  Existing methods, mainly based on localized modelling or special fixed experimental designs, will be assessed to determine the domain of problems where they are effective.  It is clear in advance, however, that new methods will be required for complex applications: those with moderate to high-dimensional input, nonlinear relationships, and/or high-order interaction effects. Only by adapting the experiment - taking further observations where the target function has special features - can a dense sampling of the input space be obtained where it matters. Divide and conquer methods are especially promising. How to divide high-dimensional space, how to choose sub-regions for data augmentation, and guidance on the number of new runs per iteration will be critical research questions here. GPs are also used indirectly in support of other machine-learning (ML) methods such as deep learning neural networks. Neural networks for image classification, for example, have "tuning" parameters that have to be set by the user, to determine the basic network architecture or regularization, for instance.  Users tune these so-called hyperparameters by trying different values and attempting to minimize validation error in various ways. To obtain the validation error requires training the ML method, which is itself computationally very intensive. Hence, systematic methods known as Bayesian optimization train a GP to model the relationship between the hyperparameter settings and validation error, and hence adaptively optimize the error. The research program will continue work in my lab on "automatic ML", to minimize the number of tries of the expensive underlying ML algorithm. Advances here will likely have impact on other computationally challenging optimization problems where the objective is produced by an expensive algorithm.

研究计划的首要目标是扩展高斯过程（GPs），以实现更复杂的应用。首先，新的方法将扩大全球定位系统，以实现大样本量，并使用自适应抽样，以便对广泛的科学和工程学科之间产生的复杂关系进行准确的统计建模。其次，使用GPs的自适应搜索方法的效率将允许自动调整计算密集型机器/统计学习器。全科医生对科学和工程产生了深远的影响，他们被直接用作机器/统计学习者。物理系统的复杂计算机代码对于优化、未知数校准、灵敏度分析等来说可能太慢。在有限的计算机模型运行中训练的GPs用于这些目的，作为特定科学目标的计算快速替代品。然而，众所周知，训练GP的计算时间随着样本量的立方而增加。因此，对于几千人或更多的样本量，全科医生的吸引力较小。现有的方法主要基于局部建模或特殊的固定实验设计，将进行评估，以确定它们在哪些问题领域是有效的。然而，很明显，复杂的应用需要新的方法：那些具有中等到高维输入、非线性关系和/或高阶交互效应的应用。只有通过调整实验——在目标函数具有特殊特征的地方进行进一步观察——才能在重要的地方获得输入空间的密集采样。分而治之的方法尤其有前途。如何划分高维空间，如何选择子区域进行数据增强，以及指导每次迭代的新运行次数将是这里的关键研究问题。全科医生也间接用于支持其他机器学习（ML）方法，如深度学习神经网络。例如，用于图像分类的神经网络具有“调谐”参数，这些参数必须由用户设置，以确定基本的网络架构或正则化。用户通过尝试不同的值并尝试以各种方式最小化验证错误来调优这些所谓的超参数。为了获得验证误差，需要训练机器学习方法，而机器学习方法本身的计算量非常大。因此，被称为贝叶斯优化的系统方法训练GP来模拟超参数设置与验证误差之间的关系，从而自适应优化误差。该研究项目将继续在我的实验室进行“自动ML”的研究，以尽量减少昂贵的底层ML算法的尝试次数。这里的进展可能会对其他具有计算挑战性的优化问题产生影响，其中目标是由昂贵的算法产生的。