Theory And Methodology For Sparse Inference

摘要：对科学研究中普遍出现的海量数据集的分析提出了许多在小规模研究中不存在的统计挑战。许多这些数据集表现出稀疏性，其中大多数数据对应于噪声，只有一小部分是感兴趣的。在这种情况下，混合模型可以为各种各样的问题提供有效和方便的框架。研究人员建议为稀疏设置下的混合模型开发一种综合方法。在中等稀疏和超稀疏环境中，有四个主要目标需要实现。第一个是精确地估计稀疏性，以及开发估计稀疏性的一般方法。第二个目标是开发一个数据依赖的阈值规则，该规则以高概率产生几乎所有与信号对应的情况集合。研究人员还计划开发一种理论，在发现信号和包含噪声之间做出精确的权衡。第三个目标是开发稀疏混合模型的最佳检测理论。最终目标是为将混合模型与序列模型连接起来提供理论基础，这是分析稀疏数据的另一个有用框架。提出的稀疏推理研究将为其他科学领域的研究人员收集和分析具有稀疏信号的大型数据集提供技术工具和方法。这些领域包括天文学、生物信息学、生物统计学和遗传学。程序和算法将在Splus或Matlab中实现，并与相关研究报告一起在互联网上提供，以便与其他方法进行比较。

ABSTRACT: The analysis of massive data sets now commonly arising in scientific investigations poses many statistical challenges not present in smaller scale studies. Many of these data sets exhibit sparsity where most of the data corresponds to noise and only a small fraction is of interest. In such situations mixture models can provide an effective and convenient framework for a wide variety of problems. The investigators propose to develop a comprehensive methodology for mixture models in sparse settings. There are four main goals to be pursued in moderately sparse and super sparse environments. The first is to make precise how well sparsity can be estimated as well as to develop a general methodology for estimating sparsity. A second goal is to develop a data dependent thresholding rule which with high probability yields a collection of cases almost all of which correspond to signal. The investigators also plan to develop a theory which makes precise the possible tradeoffs between discovering signal and including noise. A third goal is to develop a theory of optimal detection for sparse mixture models. A final goal is to provide a theoretical basis for connecting mixture models with sequence models another useful framework for analyzing sparse data.The proposed research on sparse inference will provide technical tools as well as methodology, to researchers in other scientific fields who collect and analyze large data sets with sparse signals. These fields include astronomy, bioinformatics, biostatistics, and genetics. The procedures and algorithms will be implemented in Splus or Matlab and made available on the Internet along with the associated research reports so as to facilitate comparisons with other approaches.