Rare and Weak Signals in Big Data: How to Find Them and How to Use Them

提出了一种研究努力，以创建新的工具，高维数据分析，专注于非常具有挑战性的制度，信号是罕见的和微弱的。具体而言，提议者提议： （一）. 发展了一种新的工具，用于高维变量选择的图筛选，引入了一个新的理论框架，用于评估变量选择的最优性，并显示，图筛选实现了最佳的收敛速度在汉明距离的选择错误。 （B）。利用最近提出的Higher Criticism阈值化思想，发展了一种新的谱聚类方法，并研究了与低秩矩阵恢复相关的几个问题的基本限制，包括高维聚类，稀疏主成分分析，以及与底层大尺寸协方差矩阵相关的测试问题。(c)将所提出的方法和理论扩展并应用于分析各个科学领域（包括基因组学和机器学习）中生成的大数据。我们经常说，我们正在进入“大数据”时代，在这个时代，由数百万个观察结果组成的海量数据集被挖掘出关联和模式。对于这种普遍的趋势，从来没有说过的是，不幸的是，我们正在寻找的信号通常非常罕见和微弱，很难找到，很容易被愚弄。该项目引入了适合大数据中罕见和微弱信号的新思想，新工具和新理论，并将理论和方法应用于各种科学领域，包括基因组学和机器学习。

A research effort is proposed to create new tools for high dimensional data analysis, focusing on the very challenging regime where signals are both rare and weak. In particular, the proposer proposes to: (a). Develop graphlet screening as a new tool for high dimensional variable selection, introduce a new theoretic framework for assessing the optimality of variable selection, and show that graphlet screening achieves the optimal rate of convergence in terms of Hamming distance of the selection errors. (b). Develop a new method of spectral clustering by using the recent idea of Higher Criticism thresholding, and investigates the fundamental limits for several problems related to low-rank matrix recovery, including high dimensional clustering, sparse Principle Component Analysis, and a testing problem related to the underlying large-size covariance matrix. (c) Extend and apply the proposed methods and theory to the analysis of Big data generated in various scientific fields, including genomics and machine learning. We are often said that we are entering the era of 'Big Data', where massive datasets consisting of millions of observations are mined for associations and patterns. What is never said about this pervasive trend is that, unfortunately, the signal we are looking for is usually very rare and weak and is hard to find, and it is easy to be fooled. The project introduces new ideas, new tools, and novel theory that are appropriate for rare and weak signals in Big Data, and apply the theory and methods to various scientific fields, including genomics and machine learning.