CAREER: Statistical Learning from a Modern Perspective: Over-parameterization, Regularization, and Generalization

统计方法一直是可解释、可操作和值得信赖的机器学习的主要推动力。然而，现有的统计理论在解释现代机器学习应用中出现并普遍存在的许多新现象方面仍然存在严重不足。例如，过度参数模型(即模型参数多于样本的模型)的流行挑战了我们关于偏差-方差权衡的经典统计见解；许多学习算法显示出良好的算法正则化以缓解过度拟合的事实在很大程度上超出了以前的统计文献的范围，而现代应用中风险曲线的非传统形状让许多统计学家感到困惑。然而，与为经典背景开发的丰富理论相比，这些奇怪而神秘的现象的统计基础仍然远远不够。出于这一动机，该项目的首要目标是通过使机器学习适应当代环境来丰富机器学习的统计学基础，从而将经典统计学和尖端机器学习联系起来。此外，该项目将为STEM领域中多个学科的所有级别的学生(特别是代表性不足的群体)提供宝贵的机会，并将在本文描述的任务之外的几个领域产生科学和社会影响，包括但不限于神经科学、在线教育和公平的机器学习。为了寻求可解释性和可操作的见解，该项目计划重温多个经典统计问题--从最小范数内插、风险估计、交叉验证、核增强、数据不平衡分类、转移学习--重点是揭示现代但未被探索的制度的新见解。几个反复出现的主题包括：(1)在大型模型复杂性的情况下表征精确的风险行为；(2)调和过度参数化和正则化似乎相互冲突的目标；(3)开发特定于算法的统计推理工具；以及(4)探索正则化和泛化之间的相互作用。该项目包括三个不同但又相关的推动力：(1)对过度参数化的统计见解：它探索模型复杂性和样本外性能之间的多产相互作用；(2)通过提前停止的算法正则化：旨在发展作为早期停止基础的统计原理；(3)风险(非)单调性与不平衡数据：其动机是样本量中泛化误差的非单调性，并追求原则性去偏方法来纠正它。该项目将开发一套统计洞察力，为尖端机器学习实践提供信息，以及一系列统计方法，将实际吸引现代数据驱动的应用程序。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Statistical methods have been a major driving force towards interpretable, actionable, and trustworthy machine learning. However, the existing statistical theory remains highly inadequate in explaining many new phenomena that emerge, and become pervasive in modern machine learning applications. For instance, the prevalence of over-parameterized models (i.e., the ones that have more model parameters than samples) challenges our classical statistical insights about the bias-variance tradeoff; the fact that many learning algorithms exhibit favorable algorithmic regularization to alleviate overfitting is largely beyond the reach of previous statistical literature, and the unconventional shapes of the risk curves in modern applications puzzle many statisticians. Compared to the rich theory developed for classical settings, however, the statistical underpinnings for these curious yet mysterious phenomena remain far from sufficient. Motivated by this, the overarching goal of the project is to enrich the statistical foundation of machine learning by adapting it to contemporary settings, thereby bridging classical statistics and cutting-edge machine learning. In addition, the project will provide valuable opportunities for training students (particularly underrepresented groups) at all levels across multiple disciplines in the STEM field, and will exert scientific and societal impacts on several domains beyond the tasks described herein, including but not limited to neuroscience, online education, and equitable machine learning.Striving for interpretability and actionable insights, this project plans to revisit multiple classical statistical problems---ranging from minimum-norm interpolation, risk estimation, cross validation, kernel boosting, data-imbalanced classification, to transfer learning---with an emphasis on unveiling new insights for modern yet under-explored regimes. Several recurring themes include: (i) characterizing precise risk behavior in the face of large model complexity; (ii) reconciling the seemingly conflicting goals of over-parameterization and regularization; (iii) developing algorithm-specific statistical reasoning tools; and (iv) exploring the interplay between regularization and generalization. The project comprises three distinct yet related thrusts: (1) statistical insights for over-parameterization: which explores the prolific interplay between model complexity and out-of-sample performance; (2) algorithmic regularization via early stopping: which aims to develop statistical principles that underlie early stopping; (3) risk (non)-monotonicity with imbalanced data: which is motivated by the non-monotonicity of generalization errors in the sample size and pursues principled debiasing methods to rectify it. The project will develop a suite of statistical insights that can inform cutting-edge machine learning practice, as well as an array of statistical methodologies that will be practically appealing for modern data-driven applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.