Deep Learning and Random Forests for High-Dimensional Regression

该项目旨在研究两种最广泛使用和最先进的高维回归方法：深度神经网络和随机森林。尽管它们被广泛应用，但直到最近，研究人员还没有确定它们的理论性质。这项拟议的研究旨在通过开发具有理论价值的工具，并为在工作中频繁使用这些流行方法的实践者和应用科学家提供保证和指导，来增加关于它们分析的越来越多的文献。多层网络的成功在很大程度上得益于它们的泛化能力，尽管它们能够很好地拟合大多数数据集，只要有足够的参数。当输入维度远远大于可用样本量时，这种现象尤其明显，例如在分子生物学、医学成像和天体物理学等许多现代应用中就是这种情况。提出的工作的一个主要组成部分将是获得一类深度神经网络的复杂性界限，并控制其权重的大小，然后将其用于界定泛化误差和统计风险。这些复杂性界限揭示了复杂性惩罚的作用，复杂性惩罚是基于网络权重的特定范数。在这些观察的激励下，另一项拟议的研究试图提供某些复杂性惩罚估计器及其自适应特性的统计保证。目前关于随机森林的理论结果要么是实际使用的那些结果的程式化版本，要么是本质上是渐近的，因此很难确定作为随机森林参数的函数的收敛质量。此外，分析更实际的随机森林实现的设置仅限于结构化的、固定维度的回归函数类。考虑到这些限制，该提案的第一部分旨在调查当预测因子的数量随着样本量的增加而增加时，随机森林在高维区域中的行为。另一个研究目标是分离和研究可以建立有限样本收敛速度的灵活高维回归函数族。这个项目的最后努力是将流行的不同重要性的衡量标准与随机森林的偏差联系起来。由于变量重要性度量用于评估每个预测变量在影响产出中所起的作用，这种联系将部分解释为什么随机森林适应稀疏性。这种关系还将有助于从理论上激励变量重要性度量作为模型可解释性的有用工具。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project aims to investigate two of the most widely used and state-of-the-art methods for high-dimensional regression: deep neural networks and random forests. Despite their widespread implementation, pinning down their theoretical properties has eluded researchers until recently. The proposed research aims to add to the growing body of literature on their analysis, by both developing tools of theoretical value and providing guarantees and guidance for practitioners and applied scientists who use these popular methods frequently in their work.The success of multi-layer networks has largely been buoyed by their ability to generalize well despite being able to fit most datasets, given enough parameters. This phenomenon is particularly striking when the input dimension is far greater than the available sample size, as is the case with many modern applications in molecular biology, medical imaging, and astrophysics, to name a few. A major component of the proposed work will be to obtain complexity bounds for classes of deep neural networks with controls on the size of their weights, which can then be used to bound generalization error and statistical risk. These complexity bounds reveal the role of complexity penalization, which is based on certain norms of the weights of the network. Motivated by these observations, another stream of the proposed research seeks to provide statistical guarantees of certain complexity penalized estimators and their adaptive properties. Current theoretical results for random forests are either for stylized versions of those that are used in practice or are asymptotic in nature and it is therefore difficult to determine the quality of convergence as a function of the parameters of the random forest. Furthermore, the setting for the analysis of more practical implementations of random forests is limited to structured, fixed-dimensional regression function classes. Given these restrictions, the first component of the proposal aims to investigate how random forests behave in the high-dimensional regime when the number of predictors grows with the sample size. Another research objective is to isolate and study families of flexible high-dimensional regression functions for which finite sample convergence rates can be established. The final endeavor of this project is to connect popular measures of variable importance to the bias of random forests. Since variable importance measures are used for assessing the role each predictor variable plays in influencing the output, this connection will partially explain why random forests are adaptive to sparsity. The relationship will also help to theoretically motivate variable importance measures as useful tools for model interpretability.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.