Theory and Methods for Modern Predictive Inference

在统计学和机器学习中，一个核心的推理任务是根据过去的观察结果预测未来数据的值。预测算法在日常生活中被广泛使用，如垃圾邮件检测、健康风险评估、天气预报、经济等，使其成为最基本和最重要的一类统计推理任务。在过去的十年里，深度神经网络和功能强大的计算机的出现在设计和实现预测算法方面取得了重大进展，导致了方法和应用的爆炸性发展。这些新的应用需要有数学依据的新的统计原则性方法，以充分和正确地利用这些新工具的力量。然而，算法和数据集中看不到的复杂程度对依赖于简单问题结构的经典统计和学习理论框架构成了根本挑战。在现代数据科学时代算法和计算进步的推动下，在这个研究项目中，我们计划承担几个新的方法学挑战，并填补统计预测推理中的理论空白，包括对大量预测模型的同时精度评估，以及使用校准预测进行有效的统计推理。该项目还为研究生提供了研究培训机会。本研究项目由两部分组成。第一部分将提供对交叉验证的新见解，交叉验证是最广泛使用的模型和调整参数选择方法之一。理论的发展将有助于理解交叉验证风险的联合随机性，不仅解释了广泛观察到的交叉验证的过度拟合倾向，而且指出了纠正这种趋势的潜在解决方案。这项研究工作将进一步丰富和连接多个活跃研究领域，包括交叉验证、高维高斯比较、模型置信度和在线学习。第二部分旨在通过发展一种超越互换性的基于共形的假设检验方法来填补保角预测文献中的一个重要空白。我们计划探索共形预测与经典主题之间的联系，如Mann-Whitney秩和统计量、两样本U-统计量和半参数推断。预期的结果将导致保角推理的新应用，并在这些相关研究主题中产生一系列新的研究问题。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

In statistics and machine learning, a central inferential task is to predict the values of future data given past observations. Prediction algorithms are widely used on a daily basis, such as spam detection, health risk evaluation, weather forecasting, economy, etc., making it one of the most fundamental and important classes of statistical inference tasks. In the past decade, the emergence of deep neural networks and powerful computers have made major progress in designing and implementing prediction algorithms, resulting in an explosive development of methods and applications. These new applications call for novel statistically principled methods with mathematical justifications to fully and correctly exploit the power of such new tools. However, the unseen level of complexity in both the algorithms and datasets poses fundamental challenges to classical statistical and learning-theoretical frameworks that rely on simple problem structures. Motivated by the algorithmic and computational advances in the modern data science era, in this research project, we plan to take on several new methodological challenges and fill theoretical gaps in statistical predictive inference, including simultaneous accuracy evaluation for a large collection of prediction models, and valid statistical inference using calibrated prediction. The project also provides research training opportunities for graduate students. The research project consists of two parts. The first part will provide new insights into cross-validation, one of the most widely used methods for model and tuning parameter selection. The theoretical development will contribute to the understanding of the joint randomness of cross-validated risks, not only explaining the widely observed overfitting tendency of cross-validation but also pointing out potential solutions to correct it. This research work will further enrich and connect multiple areas of active research, including cross-validation, high-dimensional Gaussian comparison, model confidence set, and online learning. The second part aims at filling an important gap in the conformal prediction literature by developing a conformal-based hypothesis testing method beyond ex-changeability. We plan to explore connections between conformal prediction and classical topics such as Mann-Whitney rank-sum statistic, two-sample U-statistics, and semiparametric inference. The expected results will lead to new applications of conformal inference and create a new array of research problems across these related research topics.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.