CIF: Small: Statistically Optimal Subsampling for Big Data and Rare Events Data

不断增加的大数据量为推进跨科学领域的知识提供了前所未有的机会。然而，传统的大数据分析涉及高计算成本，并且通常需要超级计算机。该项目旨在开发计算工具，使从业人员能够在不依赖超级计算机的情况下分析大数据。它产生最佳算法，以有限的计算资源从海量数据中提取最大量的信息。稀有事件数据在大数据中很常见，其中感兴趣的事件数量相对较少，尽管可用的完整数据很大。该项目正在确定可以在不损失任何信息的情况下丢弃大多数数据的条件，并开发对罕见事件数据进行有效分析和适当决策的方法。教育是该项目的另一个关键组成部分，重点关注课堂整合和下一代劳动力培训，旨在吸引和装备更广泛的参与者，特别是代表性不足的群体，进入计算数据科学领域。然而，现有的调查集中在收集的数据的计算，现有的结果是不适合的基础模型的统计推断。本计画在以下几个方向发展和扩充子取样技术：1）建立一个架构，借由检验子取样估计量的统计分布性质，决定统计上最佳的子取样机率：2）在一大类渐近无偏估计量中，导出最大子取样条件似然估计量，使其具有最小的渐近方差; 3）它获得了关于罕见事件数据的新的理论见解，并挑战了长期以来低估罕见事件概率的观点。该研究是对大数据子采样领域的重要补充，并提供了广泛适用于促进实际推理和决策的工具。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The ever-increasing amounts of big data offer unprecedented opportunities for advancing knowledge across scientific fields. However, traditional analyses of big data involve high computational costs and often require supercomputers. This project aims to develop computational tools that empower practitioners to analyze big data without dependency on supercomputers. It produces optimal algorithms that extract the maximum amount of information from massive data with limited computing resources. Rare-events data are common in big data where the numbers of interested events are relatively small although available full data are massive. This project is identifying conditions when the majority data can be discarded without any information loss, and developing methods for valid analysis and appropriate decision-making with rare events data. Education is another key component of the project, with a significant focus on classroom integration and next-generation workforce training, aiming to attract and equip a broader range of participants, especially underrepresented groups, to the field of computational data science.Subsampling has demonstrated a pervasive potential to enable better use of a fixed amount of computing resources. However, existing investigations focus on calculations of the collected data, and available results are not suitable for statistical inference on the underlying model. This project develops and expands the subsampling technique in the following directions: 1) It establishes a framework to determine statistically optimal subsampling probabilities by examining statistical distributional properties of subsample estimators; 2) it derives the maximum subsampled conditional likelihood estimator that has the smallest asymptotic variance among a large class of asymptotically unbiased estimators; and 3) it obtains new theoretical insights on rare-events data and challenges a long-standing view of underestimated probabilities for rare events. The research is a significant addition to the field of big data subsampling and provides tools that are widely applicable to facilitate practical inference and decision-making. It also answers important questions that are essential for extracting valid information from rare-events data.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.