New Frontiers of Robust Statistics in the Era of Big Data

现代技术促进了具有复杂结构的史无前例数量的特征的收集。尽管在从海量数据中提取有用信息方面取得了广泛的进展，但统计分析通常假定数据是在没有任何污染的情况下提取的。然而，在现实中，由于大数据时代的数据收集过程或数据的固有性质，在基因组学和医学成像等应用中产生的数据集通常更加不均匀。例如，在基因表达数据分析中，由于阵列芯片伪影，如阵列内试剂喷洒不均匀，在微阵列实验中经常会出现离群值。与大数据时代最近的进步相比，污染模型下稳健程序的建模和理论基础方面的研究已经落后。为了弥补这一差距，该项目寻求开发新的健壮的估计和推理程序，这些程序对各种污染模型是速率最优的，作为构建块来解决建模、理论和计算方面的挑战。完成后，这项工作将导致对污染模型的全面了解，并对生物学、基因组学、天文学和金融学等各个学科产生立竿见影的影响。该项目还为本科生和研究生提供培训机会，并用于丰富统计和数据科学的课程和推广教育材料。该项目旨在解决高维和非参数污染模型中的可靠程序所面临的一些最紧迫的挑战。具体地说，(I)研究开始于污染模型下的增维和高维回归中的低维参数的统计推断。PI将研究污染比例对获得根-n一致性结果的影响。文中还考虑了污染模型下的鲁棒大规模同时推理。(Ii)接下来，PI将重温任意污染分布和结构化污染分布下的一些经典的非参数密度估计问题。PI计划提出费率最优程序，并通过各种模型指标，包括污染比例、污染结构和损失函数的选择，仔细研究污染对估计的影响。(3)PI将在结构污染模型下发展一种U型稳健协方差估计，并为其率最优性提供严格的理论保证。这种一般的稳健估计器可以作为建立结构化大协方差/精度矩阵估计问题的许多速率最优过程的基础。将开发用户友好的R包来实施建议的方法。这一奖励反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Modern technologies have facilitated the collection of an unprecedented amount of features with complex structures. Although extensive progress has been made towards extracting useful information from massive data, the statistical analysis typically assumed that data are drawn without any contamination. However, in reality the data sets arising in applications such as genomics and medical imaging are usually more inhomogeneous due to either data collection process or the intrinsic nature of the data in the era of big data. For instance, in gene expression data analysis, outliers frequently arise in microarray experiments due to the array chip artifacts such as uneven spray of reagents within arrays. Compared to the recent advances in the era of big data, research in modeling and theoretical foundations for robust procedures under contamination models has fallen behind. To bridge this gap, this project seeks to develop new robust estimation and inference procedures which are rate-optimal for various contamination models as building blocks to address the modeling, theory and computational challenges. Upon completion, this work will lead to a comprehensive understanding of contamination models and have an immediate impact on various disciplines such as biology, genomics, astronomy and finance. The project also provides training opportunities for undergraduate and graduate students, and is used to enrich courses and outreach educational materials in statistics and data science.This project aims to address some of the most pressing challenges that are faced by robust procedures in high-dimensional and nonparametric contamination models. Specifically, (I) the research begins with statistical inference of low-dimensional parameters in both increasing-dimensional and high-dimensional regressions under contamination models. The PI will study the influence of contamination proportion in obtaining the root-n consistency results. Robust large-scale simultaneous inference under contamination models are also considered. (II) Next, the PI will revisit some classical nonparametric density estimation problems both under arbitrary and structured contamination distributions. The PI plans to propose rate-optimal procedures and carefully study the effect of contamination on estimation through various model indices, including contamination proportion, the structure of contamination and the choice of loss function. (III) The PI will develop a U-type robust covariance estimator under structured contamination models and provide rigorous theoretical guarantees on its rate optimality. This general robust estimator can serve as building blocks for establishing many rate-optimal procedures for structured large covariance/precision matrix estimation problems. User-friendly R packages will be developed to implement the proposed methods.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.