SHF: Small: Collaborative Research: Programming Tools for Adaptive Data Analysis

当经验研究人员根据不能概括到新数据的数据集得出结论时，就会发生错误发现或过度拟合。虽然有许多统计方法可以防止错误发现，但大多数方法都是为静态数据分析设计的，在静态数据分析中，一个数据集只使用一次。然而，现代数据分析是自适应的，通常相同的数据集被多个研究人员重复用于多项研究。适应性已经被统计学家确定为不可重复性研究的原因之一，而这个项目？S将开始解决这个问题的更广泛的意义和重要性。具体地说，该项目将建立一个原型编程工具，以防止自适应数据分析引起的错误发现。智力上的优点是将这个问题上的最新理论进展纳入并扩展到一个编程框架中，使研究人员能够自适应地分析数据集，并确保不会发生过拟合。该项目建立在差异隐私和错误发现之间令人惊讶的最近联系的基础上，错误发现是最近出现的保护敏感数据隐私的强有力的统计保证。这项工作表明，当以不同的私有方式分析数据时，就不会发生错误的发现。差分隐私也是可编程的，允许从简单的组件构建复杂的差分私有算法，因此它是自适应数据分析的理想编程框架。该项目正在将现有的不同私有编程框架扩展到自适应数据分析。PIS还在开发新的算法和编程语言工具，用于自适应数据分析，并将它们纳入这一应用的第一个原型系统。

False discovery, or overfitting, occurs when an empirical researcher draws a conclusion based on a dataset that does not generalize to new data. Although there are many statistical methods for preventing false discovery, most are designed for static data analysis, where a dataset is used only once. However, modern data analysis is adaptive, and often the same datasets are reused for multiple studies by multiple researchers. Adaptivity has been identified by statisticians as one cause of non-reproducible research, and this project?s broader significance and importance will be to begin addressing this problem. Specifically, this project will build a prototype programming tool for preventing false discovery arising from adaptive data analysis. The intellectual merits are to incorporate and extend recent theoretical advances on this problem into a programming framework that allows researchers to analyze datasets adaptively with robust guarantees that overfitting will not occur.The project builds on a surprising recent connection between differential privacy and false discovery, a robust statistical guarantee that emerged recently to protect the privacy of sensitive data. This line of work shows that when data is analyzed in a differentially private way, then false discoveries cannot occur. Differential privacy is also programmable, and allows complex differentially private algorithms to be built from simple components, so it is an ideal programming framework for adaptive data analysis. This project is extending existing differentially private programming frameworks to adaptive data analysis. The PIs are also developing new algorithmic and programming languages tools for adaptive data analysis, and incorporating them into the first prototype system for this application.