Rare and Weak Signals in Big Data: How to Find Them and How to Use Them

提出了一项研究工作，以创建用于高维数据分析的新工具，重点关注信号既罕见又弱的极具挑战性的制度。具体而言，提议人提议：(A)。发展了字形筛选作为高维变量选择的一种新工具，引入了一个新的理论框架来评估变量选择的最优性，并证明了字形筛选在选择误差的汉明距离方面达到了最优的收敛速度。(B)。利用最新的高批判阈值思想，提出了一种新的谱聚类方法，并研究了与低阶矩阵恢复有关的几个问题的基本极限，包括高维聚类、稀疏主成分分析以及与底层大尺寸协方差矩阵相关的测试问题。(C)将拟议的方法和理论推广和应用于分析各个科学领域产生的大数据，包括基因组学和机器学习。我们经常说我们正在进入大数据时代，在这个时代，由数百万个观测组成的海量数据集被挖掘出来，以寻找关联和模式。关于这种无处不在的趋势，从来没有人说过的是，不幸的是，我们寻找的信号通常非常罕见和微弱，很难找到，很容易上当。该项目引入了适用于大数据中稀有和微弱信号的新思想、新工具和新理论，并将理论和方法应用于包括基因组学和机器学习在内的各个科学领域。

A research effort is proposed to create new tools for high dimensional data analysis, focusing on the very challenging regime where signals are both rare and weak. In particular, the proposer proposes to: (a). Develop graphlet screening as a new tool for high dimensional variable selection, introduce a new theoretic framework for assessing the optimality of variable selection, and show that graphlet screening achieves the optimal rate of convergence in terms of Hamming distance of the selection errors. (b). Develop a new method of spectral clustering by using the recent idea of Higher Criticism thresholding, and investigates the fundamental limits for several problems related to low-rank matrix recovery, including high dimensional clustering, sparse Principle Component Analysis, and a testing problem related to the underlying large-size covariance matrix. (c) Extend and apply the proposed methods and theory to the analysis of Big data generated in various scientific fields, including genomics and machine learning. We are often said that we are entering the era of 'Big Data', where massive datasets consisting of millions of observations are mined for associations and patterns. What is never said about this pervasive trend is that, unfortunately, the signal we are looking for is usually very rare and weak and is hard to find, and it is easy to be fooled. The project introduces new ideas, new tools, and novel theory that are appropriate for rare and weak signals in Big Data, and apply the theory and methods to various scientific fields, including genomics and machine learning.