Theory and Methods for Modern Predictive Inference

在统计学和机器学习中，一个中心推理任务是根据过去的观察来预测未来数据的值。预测算法在日常生活中广泛使用，例如垃圾邮件检测、健康风险评估、天气预报、经济等，使其成为最基本、最重要的统计推理任务之一。在过去的十年中，深度神经网络和强大计算机的出现在预测算法的设计和实现方面取得了重大进展，导致了方法和应用的爆炸性发展。这些新应用需要具有数学依据的新颖的统计原理方法，以充分、正确地利用此类新工具的强大功能。 然而，算法和数据集中看不见的复杂程度对依赖于简单问题结构的经典统计和学习理论框架提出了根本性挑战。 受现代数据科学时代算法和计算进步的推动，在这个研究项目中，我们计划应对一些新的方法论挑战，填补统计预测推理的理论空白，包括对大量预测模型的同时准确性评估，以及使用校准预测进行有效的统计推理。该项目还为研究生提供研究培训机会。该研究项目由两部分组成。第一部分将为交叉验证提供新的见解，交叉验证是最广泛使用的模型和调整参数选择方法之一。理论发展将有助于理解交叉验证风险的联合随机性，不仅解释了广泛观察到的交叉验证的过度拟合趋势，而且指出了纠正它的潜在解决方案。这项研究工作将进一步丰富和连接多个活跃研究领域，包括交叉验证、高维高斯比较、模型置信集和在线学习。第二部分旨在通过开发一种超越可交换性的基于保形的假设检验方法来填补保形预测文献中的一个重要空白。我们计划探索保形预测与曼惠特尼秩和统计、双样本 U 统计和半参数推理等经典主题之间的联系。预期结果将带来共形推理的新应用，并在这些相关研究主题中产生一系列新的研究问题。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

In statistics and machine learning, a central inferential task is to predict the values of future data given past observations. Prediction algorithms are widely used on a daily basis, such as spam detection, health risk evaluation, weather forecasting, economy, etc., making it one of the most fundamental and important classes of statistical inference tasks. In the past decade, the emergence of deep neural networks and powerful computers have made major progress in designing and implementing prediction algorithms, resulting in an explosive development of methods and applications. These new applications call for novel statistically principled methods with mathematical justifications to fully and correctly exploit the power of such new tools. However, the unseen level of complexity in both the algorithms and datasets poses fundamental challenges to classical statistical and learning-theoretical frameworks that rely on simple problem structures. Motivated by the algorithmic and computational advances in the modern data science era, in this research project, we plan to take on several new methodological challenges and fill theoretical gaps in statistical predictive inference, including simultaneous accuracy evaluation for a large collection of prediction models, and valid statistical inference using calibrated prediction. The project also provides research training opportunities for graduate students. The research project consists of two parts. The first part will provide new insights into cross-validation, one of the most widely used methods for model and tuning parameter selection. The theoretical development will contribute to the understanding of the joint randomness of cross-validated risks, not only explaining the widely observed overfitting tendency of cross-validation but also pointing out potential solutions to correct it. This research work will further enrich and connect multiple areas of active research, including cross-validation, high-dimensional Gaussian comparison, model confidence set, and online learning. The second part aims at filling an important gap in the conformal prediction literature by developing a conformal-based hypothesis testing method beyond ex-changeability. We plan to explore connections between conformal prediction and classical topics such as Mann-Whitney rank-sum statistic, two-sample U-statistics, and semiparametric inference. The expected results will lead to new applications of conformal inference and create a new array of research problems across these related research topics.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.