Ensemble Methods for Classification/Prediction With High-Dimensional Explanatory Variables

科学和工程的进步极大地增加了可用于预测/分类感兴趣的响应结果的变量的数量。同时，数据中的信息可能是稀疏的。为了提高预测精度，提出了基于模型集成的新方法。针对具有这些特点的两个问题：复杂计算机代码的预测和药物发现数据分析中的预测/分类，将开发出方法学。 确定性计算机模型可能与高维输入(解释性)变量具有复杂的关系。例如，碳循环和植被动态的社区土地模型有数百个关于生态系统、气候、水文等的输入。约有100个变量的实验旨在进行敏感性分析，即找到对输出影响最大的输入，如总植被的测量。让数以千计的计算机模型运行是可行的，但迄今为止的工作表明，很难以有用的精度对投入产出关系进行建模。最有可能的是，输入之间存在复杂的交互影响，由于高维的原因，识别它们是一项挑战。 在药物发现中，输入变量是来自计算化学的“化学描述符”，以表征类药物分子。有许多集合可用，每个集合可以有数千个变量。响应变量或输出来自针对与疾病有关的生物靶标的活动的物理分析。将生物活性与化学投入联系起来的统计模型可以用来预测尚未被检测的分子的活性，从而提高了寻找候选药物的过程的效率。不幸的是，活性分子很少，所以响应数据中缺乏信息来适应模型。 高斯过程(GP)被广泛用于对计算机代码的确定性输入输出关系进行建模。它们还被用于药物发现数据的分析。所提出的高维输入和有限数据信息的方法是基于GP的集合，通过建立单独的模型并对它们进行平均，或者通过关联函数的集合(这是GP方法的关键)。例如，在预测准确性方面，集合具有众所周知的一般优势，并被确定为药物发现问题的最佳组合之一。他们通常通过扰动数据(自举)或划分数据观测，然后将模型拟合到创建的每个数据集来生成多个预测模型。然后，在进行预测时，对这些模型进行平均。然而，对于高维输入，响应数据中的稀疏信息意味着当模型适合数据时，大多数输入变量都不会在模型中使用。 相反，所提出的方法是在输入变量的不同子集上建立模型集合。具有交互作用的输入的子集应该在同一模型中；不交互的变量可以在不同的模型中。一次只有几个变量密集地填充数据集中的输入空间更容易，从而提高预测精度。此外，通过将变量归因于不同的模型，更多的输入有机会有助于提高预测精度。该研究计划的挑战和目标是如何识别应该在同一模型中的高维输入变量的子集，如何组合模型以获得高的整体预测精度，以及高效的算法来克服GP模型的计算需求。最重要的目标是理解像全科医生这样的统计模型应该如何调整，以适应涉及高维输入的复杂关系。

Advances in science and engineering have vastly increased the number of variables available to predict / classify a response outcome of interest. At the same time the information in the data may be sparse. Novel methods based on ensembles of models are proposed for higher prediction accuracy. Methodology will be developed for two problems with these characteristics: prediction of complex computer codes and prediction / classification in analysis of drug discovery data. Deterministic computer models can have complex relationships with high-dimensional input (explanatory) variables. For instance, the Community Land Model of the carbon cycle and vegetation dynamics has hundreds of inputs for the ecosystem, climate, hydrology, etc. Experiments with about 100 variables are aimed at sensitivity analysis, i.e., find the inputs that have most impact on an output such as a measure of total vegetation. It is feasible to make thousands of computer model runs, yet work to date shows the input-output relationships are hard to model with useful accuracy. Most likely, there are complex interaction effects between the inputs, and identifying them is a challenge because of the high dimensionality. In drug discovery, the input variables are "chemical descriptors" from computational chemistry to characterize drug-like molecules. Many sets are available, and each can have thousands of variables. The response variable or output is from a physical assay of activity against a biological target implicated in a disease. A statistical model relating biological activity to the chemical inputs can be used to predict activity of molecules that have not been assayed yet, increasing efficiency of the process to search for candidate drugs. Unfortunately, active molecules are rare, so there is a paucity of information in the response data to fit a model. Gaussian Processes (GPs) are widely used to model the deterministic input-output relationship of a computer code. They have also been used in the analysis of drug discovery data. The proposed approach to high-dimensional input and limited data information is based on ensembles of GPs, either by building separate models and averaging them, or by ensembles of correlation functions (which are key to the GP approach). Ensembles have well known general advantages in prediction accuracy and are established as among the best for the drug discovery problem, for example. They typically generate multiple prediction models by perturbing the data (bootstrapping) or dividing the data observations and then fitting a model to each data set created. The models are then averaged when making predictions. With high-dimensional input, however, sparse information in the response data means that most of the input variables are unused in a model when it is fit to data. In contrast, the proposed approach is to build an ensemble of models over distinct subsets of input variables. A subset of inputs with interaction effects should be in the same model; variables that do not interact can be in separate models. It is easier to fill the input space in a data set densely a few variables at a time, increasing prediction accuracy. Furthermore, by attributing variables to different models, more inputs have a chance to contribute to prediction accuracy. The challenges and goals of the research program are how to identify subsets of high-dimensional input variables that should be together in the same model, how to combine models for high overall prediction accuracy, and efficient algorithms to overcome the computational demands of GP models. The over-arching goal is to understand how a statistical model like a GP should be tuned to the complexities of relationships involving high-dimensional input.