CDS&E-MSS: Data thinning: methods, theory, and applications

近年来，几乎所有科学和工程领域的数据都出现了爆炸式增长。与此同时，分析这些数据的统计和机器学习方法的可用性和复杂性也随之增加。然而，在验证所得模型时出现了一个重大挑战，即确保它们捕获信号而不是噪声。在这个项目中，该项目将开发一个新的框架，将数据集分成两个独立的部分，以便一个部分可以用于拟合复杂的模型，而其余部分可以用于验证它。研究生将参与该项目，PI将努力实现统计人员的多元化。 该项目探索“数据稀疏化”，这是一种将卷积闭合分布产生的数据集分割成两个或多个独立分量的新想法，这些分量相加得出原始数据集。 PI 将数据细化框架从卷积闭合分布推广到更广泛的分布类别，方法是允许将数据集细化为两个或多个独立的组件，这些组件可以使用充分性原则，通过除加法之外的函数重新组合以生成原始数据集。一旦获得两个或多个独立组件，模型就可以适合一个组件并在其他组件上进行验证。还将探索与样本分割、数据裂变和其他相关提案的联系。 PI 还将与生物医学研究人员合作，根据不同领域的数据定制该框架，并发布高质量的软件。最终，数据细化框架将在对科学和工程中的许多问题进行模型拟合后进行严格的验证和推理。更广泛的影响包括研究生参与这项研究，以及 PI 正在进行的多样性和包容性努力。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

In recent years, there has been an explosion of data across virtually all areas of science and engineering. This has been paired with an increase in the availability and complexity of statistical and machine-learning methods to analyze these data. However, a major challenge arises in validating the resulting models, i.e. in making sure that they capture signal rather than noise. In this project, the project will develop a new framework to split up a dataset into two separate parts, so that one part can be used to fit a complex model, and the remaining part can be used to validate it. Graduate students will be involved in this project, and the PI will carry out efforts to diversify the workforce in statistics. This project explores “data thinning,” a new idea for splitting a dataset arising from a convolution-closed distribution into two or more independent components that sum to yield the original dataset. The PI will generalize the data thinning framework beyond convolution-closed distributions to a much broader class of distributions, by allowing for a dataset to be thinned into two or more independent components that can be re-combined to yield the original dataset via a function other than addition, using principles of sufficiency. Once two or more independent components have been obtained, a model can be fit to one component and validated on the other(s). Connections to sample splitting, data fission, and other related proposals will also be explored. The PI will also collaborate with biomedical researchers to tailor this framework to data from diverse domain areas and will release high-quality software. Ultimately, the data thinning framework will enable rigorous validation and inference after model fitting for a host of problems in science and engineering. Broader impacts include the involvement of graduate students in this research, as well as ongoing diversity and inclusion efforts by the PI.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.