Rare and Weak Signals in Big Data: How to Find Them and How to Use Them

大数据中的稀有信号和微弱信号：如何找到它们以及如何使用它们

• 批准号: 1208315
• 负责人: Jiashun Jin
• 金额: $ 10万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2012
• 资助国家: 美国
• 起止时间: 2012-08-01 至 2017-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1208315&HistoricalAwards=false
• 关键词: Rare; Weak; Signals; Big; Data

A research effort is proposed to create new tools for high dimensional data analysis, focusing on the very challenging regime where signals are both rare and weak. In particular, the proposer proposes to: (a). Develop graphlet screening as a new tool for high dimensional variable selection, introduce a new theoretic framework for assessing the optimality of variable selection, and show that graphlet screening achieves the optimal rate of convergence in terms of Hamming distance of the selection errors. (b). Develop a new method of spectral clustering by using the recent idea of Higher Criticism thresholding, and investigates the fundamental limits for several problems related to low-rank matrix recovery, including high dimensional clustering, sparse Principle Component Analysis, and a testing problem related to the underlying large-size covariance matrix. (c) Extend and apply the proposed methods and theory to the analysis of Big data generated in various scientific fields, including genomics and machine learning. We are often said that we are entering the era of 'Big Data', where massive datasets consisting of millions of observations are mined for associations and patterns. What is never said about this pervasive trend is that, unfortunately, the signal we are looking for is usually very rare and weak and is hard to find, and it is easy to be fooled. The project introduces new ideas, new tools, and novel theory that are appropriate for rare and weak signals in Big Data, and apply the theory and methods to various scientific fields, including genomics and machine learning.

提出了一项研究工作，以创建用于高维数据分析的新工具，重点关注信号既罕见又微弱的极具挑战性的状态。特别地，提议者建议：(a)。将石墨烯筛选作为高维变量选择的新工具，引入了评估变量选择最优性的新理论框架，并表明石墨烯筛选在选择误差的汉明距离方面达到了最优收敛速度。(b)。利用最新的高批评阈值思想，开发了一种新的光谱聚类方法，并研究了与低秩矩阵恢复相关的几个问题的基本限制，包括高维聚类，稀疏主成分分析，以及与底层大尺寸协方差矩阵相关的测试问题。(c)将提出的方法和理论扩展并应用于各种科学领域产生的大数据分析，包括基因组学和机器学习。我们常说，我们正在进入“大数据”时代，在这个时代，由数百万个观察结果组成的海量数据集被挖掘出关联和模式。不幸的是，对于这种普遍的趋势，我们从未说过的是，我们正在寻找的信号通常非常罕见和微弱，很难发现，而且很容易被愚弄。该项目引入了适合大数据中罕见和微弱信号的新思路、新工具和新理论，并将理论和方法应用于基因组学和机器学习等各个科学领域。