Ensemble Methods for Classification/Prediction With High-Dimensional Explanatory Variables

科学和工程的进步极大地增加了可用于预测/分类感兴趣的响应结果的变量的数量。同时，数据中的信息可能是稀疏的。为了提高预测精度，提出了基于模型集成的新方法。 将为具有这些特征的两个问题开发方法：复杂计算机代码的预测和药物发现数据分析中的预测/分类。确定性计算机模型可能与高维输入（解释）变量有复杂的关系。例如，碳循环和植被动态的社区土地模型有数百个生态系统、气候、水文等方面的输入，大约100个变量的实验旨在进行敏感性分析，即，找到对输出（如总植被测量）影响最大的输入。这是可行的，使成千上万的计算机模型运行，但迄今为止的工作表明，投入产出关系很难建模有用的准确性。最有可能的是，在输入之间存在复杂的交互作用效应，由于高维性，识别它们是一项挑战。在药物发现中，输入变量是来自计算化学的“化学描述符”，以表征药物样分子。有许多集合可用，每个集合可以有数千个变量。响应变量或输出来自针对疾病中涉及的生物靶标的活性的物理测定。 将生物活性与化学输入相关联的统计模型可用于预测尚未被测定的分子的活性，从而提高搜索候选药物的过程的效率。 不幸的是，活性分子很少，因此响应数据中缺乏适合模型的信息。高斯过程（GP）被广泛用于模拟计算机代码的确定性输入输出关系。它们也被用于药物发现数据的分析。所提出的高维输入和有限数据信息的方法是基于GP的集合，通过构建单独的模型并对其进行平均，或者通过相关函数的集合（这是GP方法的关键）。 集成在预测准确性方面具有众所周知的一般优势，并且被确立为例如药物发现问题的最佳方法之一。它们通常通过扰动数据（自举）或划分数据观测值，然后将模型拟合到创建的每个数据集来生成多个预测模型。 然后在进行预测时对模型进行平均。 然而，对于高维输入，响应数据中的稀疏信息意味着当模型拟合数据时，大多数输入变量在模型中未使用。 ** 相比之下，所提出的方法是在输入变量的不同子集上建立一个模型集合。 具有交互作用效应的输入子集应在同一模型中;不交互的变量可以在不同的模型中。 一次用几个变量密集地填充数据集中的输入空间更容易，从而提高预测精度。 此外，通过将变量归因于不同的模型，更多的输入有机会有助于预测的准确性。该研究计划的挑战和目标是如何识别应在同一模型中的高维输入变量的子集，如何将联合收割机模型组合以获得高的整体预测精度，以及有效的算法来克服GP模型的计算需求。 我们的首要目标是理解如何将像GP这样的统计模型调整到涉及高维输入的复杂关系。

Advances in science and engineering have vastly increased the number of variables available to predict / classify a response outcome of interest. At the same time the information in the data may be sparse. Novel methods based on ensembles of models are proposed for higher prediction accuracy. Methodology will be developed for two problems with these characteristics: prediction of complex computer codes and prediction / classification in analysis of drug discovery data.**Deterministic computer models can have complex relationships with high-dimensional input (explanatory) variables. For instance, the Community Land Model of the carbon cycle and vegetation dynamics has hundreds of inputs for the ecosystem, climate, hydrology, etc. Experiments with about 100 variables are aimed at sensitivity analysis, i.e., find the inputs that have most impact on an output such as a measure of total vegetation. It is feasible to make thousands of computer model runs, yet work to date shows the input-output relationships are hard to model with useful accuracy. Most likely, there are complex interaction effects between the inputs, and identifying them is a challenge because of the high dimensionality.**In drug discovery, the input variables are "chemical descriptors" from computational chemistry to characterize drug-like molecules. Many sets are available, and each can have thousands of variables. The response variable or output is from a physical assay of activity against a biological target implicated in a disease. A statistical model relating biological activity to the chemical inputs can be used to predict activity of molecules that have not been assayed yet, increasing efficiency of the process to search for candidate drugs. Unfortunately, active molecules are rare, so there is a paucity of information in the response data to fit a model.**Gaussian Processes (GPs) are widely used to model the deterministic input-output relationship of a computer code. They have also been used in the analysis of drug discovery data. The proposed approach to high-dimensional input and limited data information is based on ensembles of GPs, either by building separate models and averaging them, or by ensembles of correlation functions (which are key to the GP approach). Ensembles have well known general advantages in prediction accuracy and are established as among the best for the drug discovery problem, for example. They typically generate multiple prediction models by perturbing the data (bootstrapping) or dividing the data observations and then fitting a model to each data set created. The models are then averaged when making predictions. With high-dimensional input, however, sparse information in the response data means that most of the input variables are unused in a model when it is fit to data. **In contrast, the proposed approach is to build an ensemble of models over distinct subsets of input variables. A subset of inputs with interaction effects should be in the same model; variables that do not interact can be in separate models. It is easier to fill the input space in a data set densely a few variables at a time, increasing prediction accuracy. Furthermore, by attributing variables to different models, more inputs have a chance to contribute to prediction accuracy. The challenges and goals of the research program are how to identify subsets of high-dimensional input variables that should be together in the same model, how to combine models for high overall prediction accuracy, and efficient algorithms to overcome the computational demands of GP models. The over-arching goal is to understand how a statistical model like a GP should be tuned to the complexities of relationships involving high-dimensional input.