Statistical Procedures and Performance Measures for Simulator-Based Frequentist Inference

基于模拟器的频率推理的统计程序和性能测量

• 批准号: 2053804
• 负责人: Ann Lee
• 金额: $ 42.5万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-07-01 至 2024-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2053804&HistoricalAwards=false
• 关键词: Statistical; Procedures; Performance; Measures; Simulator

Many areas of the physical, engineering and biological sciences make extensive use of computer simulators to model complex systems. Whereas these simulators may be able to generate realistic synthetic data, they are often poorly suited for the inverse problem of inferring the underlying scientific mechanisms associated with observed real-world phenomena. Hence, a recent trend in the sciences has been to fit approximate models to high-fidelity simulators, and then use these approximate models for scientific inference. Inevitably, any downstream analysis will depend on the trustworthiness of the approximate model, the data collected, as well as the design of the simulations. This project will advance statistical methods for understanding complex physical systems by providing improved procedures and new performance measures for simulator-based scientific inference and uncertainty quantification. Our work will stimulate the development of data-focused collaborations, and training of students across a wide range of scientific areas, further expanding upon our ongoing interdisciplinary research efforts in high-energy physics, atmospheric science, climatology, and astronomy.Parameter estimation, confidence sets, and hypothesis testing are the hallmarks of statistical inference. Traditional methods to perform such tasks can sometimes not be applied to problems in the physical sciences because of (i) complex data settings, and (ii) the only meaningful model existing as a high-fidelity forward simulator. For example, in high-energy physics, searches of new interactions and particles require hypothesis tests involving simulations of high-dimensional collision events and their interactions with particle detectors; in cosmology, scientists regularly use large N-body simulations to understand how the Universe formed and evolved; and in atmospheric science, inferring land-air carbon fluxes based on satellite observations relies on complex atmospheric transport models. A key question is whether one can still construct hypothesis tests and confidence sets with proper frequentist coverage and high power when the likelihood function, which connects underlying parameters with observable data, is intractable but one can forward-simulate observable data from an implicit likelihood model. A related question is how to calibrate and assess the performance of surrogate models fit to high-fidelity simulations. This project works toward designing statistical procedures that unify classical statistics with modern machine learning (e.g., deep generative models, neural network classifiers and convex optimization) via the following aims: (1) Scalable tools and theory for constructing statistical tests and frequentist confidence sets with finite-sample validity in a simulator-based inference setting; (2) Statistically rigorous validation methods, which can quantify and diagnose the quality of fitted models of high-dimensional data with statistical confidence across both feature and parameter space; and (3) Sequential testing strategies that allow us to identify how to best simulate data to improve tests and confidence sets in Aim 1.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

物理、工程和生物科学的许多领域广泛使用计算机模拟器来模拟复杂系统。虽然这些模拟器可能能够生成真实的合成数据，但它们通常不适合推断与观察到的现实世界现象相关的潜在科学机制的逆问题。因此，科学界最近的一个趋势是将近似模型拟合到高保真模拟器上，然后使用这些近似模型进行科学推理。不可避免的是，任何下游分析都将取决于近似模型的可信度、收集的数据以及模拟的设计。该项目将通过为基于模拟器的科学推理和不确定性量化提供改进的程序和新的性能措施，推进理解复杂物理系统的统计方法。我们的工作将促进以数据为中心的合作的发展，并在广泛的科学领域培训学生，进一步扩大我们在高能物理，大气科学，气候学和天文学方面正在进行的跨学科研究工作。参数估计，置信度集和假设检验是统计推断的标志。执行这些任务的传统方法有时不能应用于物理科学中的问题，因为（i）复杂的数据设置，以及（ii）唯一有意义的模型作为高保真正演模拟器存在。例如，在高能物理学中，寻找新的相互作用和粒子需要进行假设检验，包括模拟高维碰撞事件及其与粒子探测器的相互作用;在宇宙学中，科学家经常使用大型N体模拟来了解宇宙如何形成和演变;在大气科学中，根据卫星观测推断陆地-空气碳通量依赖于复杂的大气传输模型。一个关键的问题是，是否仍然可以构建假设检验和置信度集与适当的频率覆盖率和高功率时的似然函数，它连接的基础参数与可观察的数据，是棘手的，但可以向前模拟可观察的数据从隐式似然模型。一个相关的问题是如何校准和评估代理模型的性能适合高保真模拟。该项目致力于设计将经典统计与现代机器学习相统一的统计程序（例如，深度生成模型、神经网络分类器和凸优化）通过以下目标：（1）用于在基于模拟器的推理设置中构造具有有限样本有效性的统计测试和频率置信集的可扩展工具和理论;（2）统计上严格的验证方法，它可以量化和诊断高维数据的拟合模型的质量，具有跨特征和参数空间的统计置信度;和（3）顺序测试策略，使我们能够确定如何最好地模拟数据，以改善目标1中的测试和置信度集。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估来支持。

项目成果

Online Platt Scaling with Calibeating

• DOI: 10.48550/arxiv.2305.00070
• 发表时间: 2023-04
• 作者: Chirag Gupta;Aaditya Ramdas

Top-label calibration and multiclass-to-binary reductions

• 发表时间: 2021-07
• 作者: Chirag Gupta;Aaditya Ramdas

Statistical constraints on climate model parameters using a scalable cloud-based inference framework

使用可扩展的基于云的推理框架对气候模型参数进行统计约束

• DOI: 10.1017/eds.2023.12
• 发表时间: 2023
• 期刊: Environmental Data Science
• 作者: Carzon, James;Abreu, Bruno;Regayre, Leighton;Carslaw, Kenneth;Deaconu, Lucia;Stier, Philip;Gordon, Hamish;Kuusela, Mikael
• 通讯作者: Kuusela, Mikael

Simulator-Based Inference with WALDO: Confidence Regions by Leveraging Prediction Algorithms and Posterior Estimators for Inverse Problems

• 发表时间: 2022-05
• 作者: Luca Masserano;T. Dorigo;Rafael Izbicki;Mikael Kuusela;Ann B. Lee

A unified framework for bandit multiple testing

• 发表时间: 2021-07
• 期刊: ArXiv
• 作者: Ziyu Xu;Ruodu Wang;Aaditya Ramdas