Theory and Methods for Modern Predictive Inference

在统计学和机器学习中，一个核心的推理任务是根据过去的观察结果预测未来数据的值。预测算法在日常生活中被广泛使用，如垃圾邮件检测、健康风险评估、天气预报、经济等，使其成为统计推理任务中最基本和最重要的一类。在过去的十年中，深度神经网络和强大的计算机的出现在设计和实现预测算法方面取得了重大进展，导致了方法和应用的爆炸式发展。这些新的应用需要新颖的统计原则方法和数学证明，以充分和正确地利用这些新工具的力量。然而，算法和数据集中看不见的复杂性对依赖于简单问题结构的经典统计和学习理论框架提出了根本性的挑战。在现代数据科学时代的算法和计算进步的激励下，在本研究项目中，我们计划承担几个新的方法挑战，填补统计预测推断的理论空白，包括对大量预测模型的同时精度评估，以及使用校准预测进行有效的统计推断。该项目还为研究生提供研究培训机会。本研究项目由两部分组成。第一部分将提供对交叉验证的新见解，交叉验证是模型和调优参数选择中最广泛使用的方法之一。理论的发展将有助于理解交叉验证风险的联合随机性，不仅解释了广泛观察到的交叉验证的过拟合趋势，而且指出了纠正它的潜在解决方案。这项研究工作将进一步丰富和连接多个活跃的研究领域，包括交叉验证、高维高斯比较、模型置信集和在线学习。第二部分旨在通过发展一种超越互换性的基于保形的假设检验方法来填补保形预测文献的重要空白。我们计划探索保形预测与经典主题（如Mann-Whitney秩和统计、双样本u统计和半参数推理）之间的联系。预期的结果将导致保形推理的新应用，并在这些相关的研究课题中产生一系列新的研究问题。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

In statistics and machine learning, a central inferential task is to predict the values of future data given past observations. Prediction algorithms are widely used on a daily basis, such as spam detection, health risk evaluation, weather forecasting, economy, etc., making it one of the most fundamental and important classes of statistical inference tasks. In the past decade, the emergence of deep neural networks and powerful computers have made major progress in designing and implementing prediction algorithms, resulting in an explosive development of methods and applications. These new applications call for novel statistically principled methods with mathematical justifications to fully and correctly exploit the power of such new tools. However, the unseen level of complexity in both the algorithms and datasets poses fundamental challenges to classical statistical and learning-theoretical frameworks that rely on simple problem structures. Motivated by the algorithmic and computational advances in the modern data science era, in this research project, we plan to take on several new methodological challenges and fill theoretical gaps in statistical predictive inference, including simultaneous accuracy evaluation for a large collection of prediction models, and valid statistical inference using calibrated prediction. The project also provides research training opportunities for graduate students. The research project consists of two parts. The first part will provide new insights into cross-validation, one of the most widely used methods for model and tuning parameter selection. The theoretical development will contribute to the understanding of the joint randomness of cross-validated risks, not only explaining the widely observed overfitting tendency of cross-validation but also pointing out potential solutions to correct it. This research work will further enrich and connect multiple areas of active research, including cross-validation, high-dimensional Gaussian comparison, model confidence set, and online learning. The second part aims at filling an important gap in the conformal prediction literature by developing a conformal-based hypothesis testing method beyond ex-changeability. We plan to explore connections between conformal prediction and classical topics such as Mann-Whitney rank-sum statistic, two-sample U-statistics, and semiparametric inference. The expected results will lead to new applications of conformal inference and create a new array of research problems across these related research topics.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.