AF: Medium: Collaborative Research: Foundations of Adaptive Data Analysis

严格分析数据的经典工具假设分析是静态的：要测试的模型和假设是独立于数据的固定的，数据的初步分析不会反馈到数据收集过程中。另一方面，现代数据分析具有很强的适应性。现代机器学习的很大一部分通过迭代调整超参数来执行作为数据函数的模型选择，并进行探索性数据分析以建议假设，然后在用于发现假设的相同数据集上进行验证。这种适应性通常被称为p黑客，并在一定程度上被指责为某些实验领域中不可复制的科学惊人流行的原因。该项目旨在开发严格的工具和方法，以在自适应环境下执行统计有效的数据分析，借鉴统计学、信息论、差异隐私和稳定算法设计的技术。该项目的技术目标包括：1)信息论方法，描述最坏情况下的数据分析在与数据集相互作用的情况下可能过度拟合的程度；2)超越最坏情况设置的数据分析师模型；以及3)弥合理论与实践之间差距的实证调查。自适应数据分析问题(也称为选择后推理，或选择性推理)在过去几年里引起了计算机科学和统计学的关注，但却来自相对分散的社区。该项目的部分目标是将这两项工作结合起来。这个项目的研究团队跨越了计算机科学、统计学和生物医学数据科学等多个部门。除了试图统一这两个领域，这项研究的更广泛的影响将是使科学更可靠，并减少“过度匹配”和“错误发现”的流行。该项目也有一个重要的外展和教育部分，将教育研究生，组织研讨会，并制作说明性材料。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Classical tools for rigorously analyzing data make the assumption that the analysis is static: the models and the hypotheses to be tested are fixed independently of the data, and preliminary analysis of the data does not feed back into the data gathering procedure. On the other hand, modern data analysis is highly adaptive. Large parts of modern machine learning perform model selection as a function of the data by iteratively tuning hyper-parameters, and exploratory data analysis is conducted to suggest hypotheses, which are then validated on the same data sets used to discover them. This kind of adaptivity is often referred to as p-hacking, and blamed in part for the surprising prevalence of non-reproducible science in some empirical fields. This project aims to develop rigorous tools and methodologies to perform statistically valid data analysis in the adaptive setting, drawing on techniques from statistics, information theory, differential privacy, and stable algorithm design. The technical goals of this project include coming up with: 1) information-theoretic measures that characterize the degree to which a worst-case data analysis can over-fit, given an interaction with a dataset; 2) models for data analysts that move beyond the worst-case setting, and; 3) empirical investigations that bridge the gap between theory and practice. The problem of adaptive data analysis (also called post-selection inference, or selective inference) has attracted attention in both computer science and statistics over the past several years, but from relatively disjoint communities. Part of the aim of this project is to integrate these two lines of work. The team of researchers on this project span departments of computer science, statistics, and biomedical data science. In addition to attempting to unify these two areas, the broader impacts of this research will be to make science more reliable, and reduce the prevalence of "over-fitting" and "false discovery." The project also has a significant outreach and education component, and will educate graduate students, organize workshops, and produce expository materials.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.