Collaborative Research: Fine-Grained Statistical Inference in High Dimension: Actionable Information, Bias Reduction, and Optimality

新兴的数据科学应用需要从庞大而混乱的数据集中有效地提取可操作的见解。相关特征的数量通常会超过可用的数据量，这会使统计推断任务和随后的决策变得非常复杂。在现有的统计文献中，大多数理论的目的是理解高维统计估计量的平均或全局行为。然而，在许多应用中，通常情况下，目标不是探索参数估计器的全局行为，而是对其局部但重要的操作属性进行推断和推理。 该项目中开发的技术和方法将进一步推动广泛领域之间的相互作用，包括高维统计，谐波分析，统计物理，优化，复杂分析和统计机器学习。该项目为研究生提供了研究培训机会。该项目追求细粒度的推理程序和理论，旨在扩大各种高维低复杂度模型的不确定性评估工具箱。本研究主要围绕以下几个典型问题展开：（1）构造低秩矩阵估计中特征向量线性泛函的最优置信区间;（2）设计一般设计下稀疏回归的细粒度假设检验方法;（3）发展缺失数据下主成分分析的逐项推断方法;（4）在最小特征间隙下进行可靠的自适应统计特征分析。重点放在模型不可知和完全适应数据异方差的算法上。解决这些问题需要发展新的统计理论，从而能够可靠地推断出未知参数下的广泛的局部属性。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估来支持。

Emerging data science applications require efficient extraction of actionable insights from large and messy datasets. The number of relevant features often overwhelms the volume of data that is available, which dramatically complicates the statistical inference tasks and subsequent decision making. In the existing statistical literature, most of theory aims at understanding the average or global behavior of a statistical estimator in high dimensions. In many applications, however, it is often the case that the goal is not to explore the global behavior of a parameter estimator, but rather to perform inference and reasoning on its local, yet important, operational properties. The techniques and methods developed in the project will further advance the interplay between a broad range of areas including high-dimensional statistics, harmonic analysis, statistical physics, optimization, complex analysis, and statistical machine learning. The project provides research training opportunities for graduate students.This project pursues fine-grained inferential procedures and theory, aimed at enlarging the uncertainty assessment toolbox for various low-complexity models in high dimensions. Focusing on a few stylized problems, this research program consists of four major thrusts: (1) construct optimal confidence intervals for linear functionals of eigenvectors in low-rank matrix estimation; (2) design fine-grained hypothesis testing procedures for sparse regression under general designs; (3) develop entry-wise inference schemes for principal component analysis with missing data; and (4) conduct reliable and adaptive statistical eigen-analysis under minimal eigen-gaps. Emphasis is placed on algorithms that are model-agnostic and fully adaptive to data heteroscedasticity. Addressing these issues calls for the development of new statistical theory that enables reliable inference for a broad class of local properties underlying the unknown parameters.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.