BIGDATA: Collaborative Research: F: Statistical Theory and Methods Beyond the Dimensionality Barrier

随着最近技术的进步，现在可以测量和记录单个个体的大量特征。大数据的数量、速度和种类，即“3V”，给这些海量数据集的建模和分析带来了巨大的挑战。例如，为了在基因水平上理解癌症，研究人员需要从从有限数量的受试者那里获得的数千、甚至数百万个候选遗传标记中检测出罕见而微弱的信号。现有的方法通常假设受试者的数量非常大，这一假设在实践中经常被违反。这个项目的主要目标是为极大的维度、小样本的数据开发有效的方法。这些方法上的进步将在应对不同领域的大数据挑战方面具有极其宝贵的价值，例如医学研究、生物信息学、金融分析和天文图像分析。这项研究的关键创新想法是从一个新的布局角度来看待高维问题，它允许变量p的数量任意大，而观测的数量n有限。本研究将系统地研究这一“有限n，任意大p”范式下的三个基本问题：(1)伪相关的渐近理论，(2)低阶相关结构的快速检测，以及(3)检测稀有和微弱信号的检测边界和最优测试方法。本研究将改变目前的渐近框架，从“大n，小p”和“大n，大p”的体制过渡到“有限n，任意大p”的体制。

With recent advances in technology, it is now possible to measure and record significant numbers of features on a single individual. The volume, velocity, and variety, the "3Vs", of Big Data pose significant challenges for modeling and analysis of these massive datasets. For example, to understand cancer at the genetic level, researchers need to detect rare and weak signals from thousands, or even millions, of candidate genetic markers obtained from a limited number of subjects. Existing methods typically assume that the number of subjects is very large, an assumption often violated in practice. The main goal of this project is to develop efficient methods for extremely large-dimensional, small sample size data. The methodological advances will be extremely valuable in addressing Big Data challenges in different areas such as medical research, bioinformatics, financial analysis, and astronomic image analysis. Efficient software packages and algorithms to implement the proposed methods will be developed and made publicly available.The key innovative idea motivating this research is viewing a high-dimensional problem from a novel packing perspective, which allows the number of variables, p, to be arbitrarily large and the number of observations, n, to be finite. The proposed research will systematically investigate three fundamental problems under this "finite n, arbitrarily large p" paradigm: (1) asymptotic theory of spurious correlations, (2) fast detection of low-rank correlation structures, and (3) detection boundary and optimal testing procedures for detecting rare and weak signals. This research will transform the current asymptotic framework, transitioning from the regimes of "large n, small p" and "large n, larger p" to the regime of "finite n, arbitrarily large p".