BIGDATA: Collaborative Research: F: Statistical Theory and Methods Beyond the Dimensionality Barrier

随着技术的最新进步，现在可以测量和记录单个人的大量特征。 大数据的体积、速度和多样性，即“3V”，对这些海量数据集的建模和分析提出了重大挑战。 例如，为了在遗传水平上了解癌症，研究人员需要从有限数量的受试者中获得的数千甚至数百万个候选遗传标记中检测到罕见和微弱的信号。 现有的方法通常假设受试者的数量非常大，这是一个在实践中经常违反的假设。 该项目的主要目标是为非常大的维度，小样本量的数据开发有效的方法。 方法上的进步对于解决医学研究、生物信息学、金融分析和天文图像分析等不同领域的大数据挑战将是非常有价值的。 高效的软件包和算法来实现所提出的方法将被开发和公开，激励这项研究的关键创新思想是从一个新的包装的角度来看，它允许变量的数量，p，是任意大的和观察的数量，n，是有限的高维问题。 本研究将系统地研究在这种“有限n，任意大p”范式下的三个基本问题：（1）伪相关的渐近理论，（2）低秩相关结构的快速检测，以及（3）检测边界和检测稀有和弱信号的最佳测试程序。 这项研究将改变目前的渐近框架，从“大n，小p”和“大n，大p”的制度过渡到“有限n，任意大p”的制度。

With recent advances in technology, it is now possible to measure and record significant numbers of features on a single individual. The volume, velocity, and variety, the "3Vs", of Big Data pose significant challenges for modeling and analysis of these massive datasets. For example, to understand cancer at the genetic level, researchers need to detect rare and weak signals from thousands, or even millions, of candidate genetic markers obtained from a limited number of subjects. Existing methods typically assume that the number of subjects is very large, an assumption often violated in practice. The main goal of this project is to develop efficient methods for extremely large-dimensional, small sample size data. The methodological advances will be extremely valuable in addressing Big Data challenges in different areas such as medical research, bioinformatics, financial analysis, and astronomic image analysis. Efficient software packages and algorithms to implement the proposed methods will be developed and made publicly available.The key innovative idea motivating this research is viewing a high-dimensional problem from a novel packing perspective, which allows the number of variables, p, to be arbitrarily large and the number of observations, n, to be finite. The proposed research will systematically investigate three fundamental problems under this "finite n, arbitrarily large p" paradigm: (1) asymptotic theory of spurious correlations, (2) fast detection of low-rank correlation structures, and (3) detection boundary and optimal testing procedures for detecting rare and weak signals. This research will transform the current asymptotic framework, transitioning from the regimes of "large n, small p" and "large n, larger p" to the regime of "finite n, arbitrarily large p".