Collaborative Research: Fine-Grained Statistical Inference in High Dimension: Actionable Information, Bias Reduction, and Optimality

新兴的数据科学应用需要从庞大而杂乱的数据集中高效地提取可操作的见解。相关特征的数量往往超过可用的数据量，这极大地使统计推断任务和随后的决策变得复杂。在现有的统计文献中，大多数理论的目的是在高维上理解统计估计量的平均或全局行为。然而，在许多应用中，通常的情况是，目标不是探索参数估计器的全局行为，而是对其局部但重要的运算性质进行推理和推理。该项目开发的技术和方法将进一步促进高维统计、调和分析、统计物理、优化、复杂分析和统计机器学习等广泛领域之间的相互作用。该项目为研究生提供了研究培训的机会，该项目追求细粒度的推理过程和理论，旨在扩大各种高维低复杂性模型的不确定性评估工具箱。针对几个典型的问题，本研究包括四个主要工作：(1)构造低阶矩阵估计中特征向量的线性泛函的最优置信度区间；(2)在一般设计下设计稀疏回归的细粒度假设检验步骤；(3)开发具有缺失数据的主成分分析的入口式推理方案；(4)在最小特征间隔下进行可靠和自适应的统计特征分析。重点放在模型不可知和完全适应数据异方差的算法上。解决这些问题需要开发新的统计理论，以实现对潜在未知参数的广泛类别的本地属性的可靠推断。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Emerging data science applications require efficient extraction of actionable insights from large and messy datasets. The number of relevant features often overwhelms the volume of data that is available, which dramatically complicates the statistical inference tasks and subsequent decision making. In the existing statistical literature, most of theory aims at understanding the average or global behavior of a statistical estimator in high dimensions. In many applications, however, it is often the case that the goal is not to explore the global behavior of a parameter estimator, but rather to perform inference and reasoning on its local, yet important, operational properties. The techniques and methods developed in the project will further advance the interplay between a broad range of areas including high-dimensional statistics, harmonic analysis, statistical physics, optimization, complex analysis, and statistical machine learning. The project provides research training opportunities for graduate students.This project pursues fine-grained inferential procedures and theory, aimed at enlarging the uncertainty assessment toolbox for various low-complexity models in high dimensions. Focusing on a few stylized problems, this research program consists of four major thrusts: (1) construct optimal confidence intervals for linear functionals of eigenvectors in low-rank matrix estimation; (2) design fine-grained hypothesis testing procedures for sparse regression under general designs; (3) develop entry-wise inference schemes for principal component analysis with missing data; and (4) conduct reliable and adaptive statistical eigen-analysis under minimal eigen-gaps. Emphasis is placed on algorithms that are model-agnostic and fully adaptive to data heteroscedasticity. Addressing these issues calls for the development of new statistical theory that enables reliable inference for a broad class of local properties underlying the unknown parameters.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.