CIF: Small: Statistically Optimal Subsampling for Big Data and Rare Events Data

不断增长的大数据量为跨科学领域的知识发展提供了前所未有的机会。然而，传统的大数据分析涉及高计算成本，通常需要超级计算机。该项目旨在开发计算工具，使从业者能够在不依赖超级计算机的情况下分析大数据。它产生最优算法，在有限的计算资源下从海量数据中提取最大数量的信息。稀有事件数据在大数据中很常见，在大数据中，尽管可用的完整数据非常大，但感兴趣的事件数量相对较少。该项目正在确定在不丢失任何信息的情况下可以丢弃大部分数据的条件，并开发针对罕见事件数据进行有效分析和适当决策的方法。教育是该项目的另一个关键组成部分，重点关注课堂整合和下一代劳动力培训，旨在吸引和装备更广泛的参与者，特别是代表性不足的群体，进入计算数据科学领域。子抽样已经证明了一种普遍的潜力，可以更好地利用固定数量的计算资源。然而，现有的研究主要集中在收集数据的计算上，现有的结果不适合对基础模型进行统计推断。本项目从以下几个方面发展和扩展了子抽样技术：1)通过检查子抽样估计量的统计分布特性，建立了一个框架来确定统计上最优的子抽样概率；2)在一大类渐近无偏估计量中，导出渐近方差最小的最大下抽样条件似然估计量；3)它获得了关于罕见事件数据的新的理论见解，并挑战了长期以来低估罕见事件概率的观点。该研究是对大数据子抽样领域的重要补充，并为促进实际推理和决策提供了广泛适用的工具。它还回答了从罕见事件数据中提取有效信息所必需的重要问题。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The ever-increasing amounts of big data offer unprecedented opportunities for advancing knowledge across scientific fields. However, traditional analyses of big data involve high computational costs and often require supercomputers. This project aims to develop computational tools that empower practitioners to analyze big data without dependency on supercomputers. It produces optimal algorithms that extract the maximum amount of information from massive data with limited computing resources. Rare-events data are common in big data where the numbers of interested events are relatively small although available full data are massive. This project is identifying conditions when the majority data can be discarded without any information loss, and developing methods for valid analysis and appropriate decision-making with rare events data. Education is another key component of the project, with a significant focus on classroom integration and next-generation workforce training, aiming to attract and equip a broader range of participants, especially underrepresented groups, to the field of computational data science.Subsampling has demonstrated a pervasive potential to enable better use of a fixed amount of computing resources. However, existing investigations focus on calculations of the collected data, and available results are not suitable for statistical inference on the underlying model. This project develops and expands the subsampling technique in the following directions: 1) It establishes a framework to determine statistically optimal subsampling probabilities by examining statistical distributional properties of subsample estimators; 2) it derives the maximum subsampled conditional likelihood estimator that has the smallest asymptotic variance among a large class of asymptotically unbiased estimators; and 3) it obtains new theoretical insights on rare-events data and challenges a long-standing view of underestimated probabilities for rare events. The research is a significant addition to the field of big data subsampling and provides tools that are widely applicable to facilitate practical inference and decision-making. It also answers important questions that are essential for extracting valid information from rare-events data.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.