New Frontiers of Robust Statistics in the Era of Big Data

现代技术促进了具有复杂结构的前所未有数量的特征的收集。虽然在从大量数据中提取有用信息方面取得了广泛的进展，但统计分析通常假设数据是在没有任何污染的情况下提取的。然而，在现实中，在基因组学和医学成像等应用中产生的数据集通常由于数据收集过程或大数据时代数据的内在性质而更加不均匀。例如，在基因表达数据分析中，由于阵列芯片伪像（如阵列内试剂的不均匀喷射），在微阵列实验中经常出现离群值。与大数据时代的最新进展相比，污染模型下鲁棒程序的建模和理论基础研究已经落后。为了弥合这一差距，该项目旨在开发新的鲁棒估计和推理程序，这些程序对于各种污染模型来说是速率最优的，作为构建模块，以解决建模，理论和计算挑战。 完成后，这项工作将导致对污染模型的全面了解，并对生物学，基因组学，天文学和金融等各个学科产生直接影响。该项目还为本科生和研究生提供培训机会，并用于丰富统计和数据科学方面的课程和宣传教育材料，旨在解决高维和非参数污染模型中稳健程序所面临的一些最紧迫的挑战。具体而言，（1）研究从污染模型下的增维和高维回归中的低维参数的统计推断开始。PI将研究污染比例对获得root-n一致性结果的影响。也考虑了污染模型下的鲁棒大规模同时推理。(II)接下来，PI将在任意和结构化污染分布下重新讨论一些经典的非参数密度估计问题。PI计划提出速率优化程序，并通过各种模型指标（包括污染比例、污染结构和损失函数的选择）仔细研究污染对估计的影响。(III)PI将在结构化污染模型下建立一个U型稳健协方差估计器，并对其速率最优性提供严格的理论保证。这个一般的鲁棒估计器可以作为构建模块，用于建立结构化大协方差/精度矩阵估计问题的许多速率最优程序。该奖项反映了NSF的法定使命，并被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

Modern technologies have facilitated the collection of an unprecedented amount of features with complex structures. Although extensive progress has been made towards extracting useful information from massive data, the statistical analysis typically assumed that data are drawn without any contamination. However, in reality the data sets arising in applications such as genomics and medical imaging are usually more inhomogeneous due to either data collection process or the intrinsic nature of the data in the era of big data. For instance, in gene expression data analysis, outliers frequently arise in microarray experiments due to the array chip artifacts such as uneven spray of reagents within arrays. Compared to the recent advances in the era of big data, research in modeling and theoretical foundations for robust procedures under contamination models has fallen behind. To bridge this gap, this project seeks to develop new robust estimation and inference procedures which are rate-optimal for various contamination models as building blocks to address the modeling, theory and computational challenges. Upon completion, this work will lead to a comprehensive understanding of contamination models and have an immediate impact on various disciplines such as biology, genomics, astronomy and finance. The project also provides training opportunities for undergraduate and graduate students, and is used to enrich courses and outreach educational materials in statistics and data science.This project aims to address some of the most pressing challenges that are faced by robust procedures in high-dimensional and nonparametric contamination models. Specifically, (I) the research begins with statistical inference of low-dimensional parameters in both increasing-dimensional and high-dimensional regressions under contamination models. The PI will study the influence of contamination proportion in obtaining the root-n consistency results. Robust large-scale simultaneous inference under contamination models are also considered. (II) Next, the PI will revisit some classical nonparametric density estimation problems both under arbitrary and structured contamination distributions. The PI plans to propose rate-optimal procedures and carefully study the effect of contamination on estimation through various model indices, including contamination proportion, the structure of contamination and the choice of loss function. (III) The PI will develop a U-type robust covariance estimator under structured contamination models and provide rigorous theoretical guarantees on its rate optimality. This general robust estimator can serve as building blocks for establishing many rate-optimal procedures for structured large covariance/precision matrix estimation problems. User-friendly R packages will be developed to implement the proposed methods.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.