Towards Efficient Bias Correction in Data Snooping

统计模型的选择通常是数据分析的关键部分，因为一个有用的模型可以帮助研究人员从嘈杂的数据中提取相关信息，以获得可解释的结果。 虽然科学或经济理论确实有助于在某些应用中建立模型，但大多数数据分析师必须依赖经验模型。 使用相同的数据来选择模型，然后执行基于模型的统计推断通常被称为数据窥探。 不幸的是，如果不仔细分析这种做法所导致的潜在偏见，数据窥探本质上是有风险的。 本项目的主要目标是研究如何理解和纠正数据窥探的偏差，并开发合理的统计推断方法。 该研究将为依赖数据驱动模型进行不确定性评估和验证性数据分析的科学家、研究人员和政策制定者提供有价值的工具。该项目侧重于对治疗效果的回归调整推断和对最佳选择亚组的推断。拟议的工作的动机是迫切需要更多的基础研究有关的处理“后选择偏见”的统计分析。重复的数据分割方法，用于对结构参数（例如，平均治疗效果）进行去偏推断，能够有效地消除解决内在科学问题的偏倚。 对最佳选择亚组的推断提供了对亚组效应大小的自然估计的偏差校正，因此降低了亚组分析中数据窥探和错误发现的风险。 在大数据时代，数据驱动模型和子组分析通常用于利用数据结构中预期的稀疏性或探索数据异构性。拟议的研究旨在为此类努力中更明智的决策提供见解，理论和工具。该项目将涉及与调查脑震荡风险的研究人员以及生物技术行业的科学家的合作，这些科学家通常依赖于亚组分析。 该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The choice of a statistical model is often a critical part of data analysis because a useful model helps researchers extract relevant information from noisy data to reach interpretable findings. While scientific or economic theories do help formulate models in some applications, most data analysts have to rely on empirical models. Using the same data to select a model and then to perform model-based statistical inference is commonly known as data snooping. Unfortunately, data snooping is intrinsically risky without a careful analysis of the potential bias resulting from such practices. The primary goal of this project is to study how to understand and correct bias from data snooping and develop sound statistical inference methods. The research will provide valuable tools for scientists, researchers, and policy makers who rely on data-driven models for uncertainty assessment and confirmatory data analysis.This project focuses on regression-adjusted inference on treatment effects and inference on the best selected subgroup. The proposed work is motivated by the pressing need for more fundamental research related to the handling of "post-selection bias" in statistical analysis. The repeated data-splitting method for de-biased inference on a structural parameter (for example, the average treatment effect) enables efficient bias removal in addressing an intrinsic scientific question. The proposed inference on the best selected subgroup provides a bias-correction to a natural estimate of the subgroup effect size, and therefore reduces the risk of data-snooping and false discoveries in subgroup analysis. In the big data era, data-driven models and subgroup analyses are often used to take advantage of anticipated sparsity in the data structure or to explore data heterogeneity. The proposed research aims to provide insights, theory, and tools for more informed decision making in such endeavors. The project will involve collaborations with researchers investigating the risk of concussion as well as scientists in the biotechnology industry who routinely rely on subgroup analysis. Graduate and undergraduate students will be engaged in the proposed research.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.