Analysis of High Dimensional Data Using Subspace Clustering

当今经验科学的一个重要而明显的趋势是，包含数千到数亿个测量值的大型数据集日益流行。例子包括高通量测量技术产生的数据集，如基因表达阵列、蛋白质组学和计算机网络监测。虽然大数据集的分析对科学家来说很重要，但它通常超出了经典统计方法的范围，并且经常提出新的概念和计算挑战。受资助的研究有两个主要部分。首先，研究人员正在研究数据挖掘领域的一个相对较新的发展，即子空间聚类，在高维数据的探索性统计分析中的应用。在第二部分中，研究人员将统计学和概率论的思想应用于新的子空间聚类方法的开发，并对其结果进行严格的数学分析。研究是在与生物科学家持续合作的背景下进行的，并且正在被纳入合作科学家将用于识别和评估各种大型数据集中重要样本变量关联的软件中。当今经验科学的一个重要而明显的趋势是，包含数千到数亿个测量值的大型数据集日益突出。例子包括高通量测量技术产生的数据集，如基因表达阵列、蛋白质组学和计算机网络监测。虽然小到中等规模的数据集通常具有比测量值更多的样本，但在大型数据集中，通常具有比样本更多的测量值，即所谓的“高维低样本量”。研究人员正在研究被称为子空间聚类的数据挖掘方法在高维数据探索性分析中的应用。子空间聚类识别给定数据矩阵中不同的样本变量相互作用（子矩阵）。与标准的双向聚类不同，不同聚类的样本和变量集可以重叠。研究人员正在研究现有子空间聚类算法的噪声敏感性，并正在开发和实现新的基于平均选择标准的子空间聚类方法，这些方法更适合于存在噪声的应用。作为这些方法的一种应用，它们利用子空间聚类对高维数据进行分类。使用组合概率的各种工具，研究人员也在开发一个严格的理论框架，其中多重测试和子空间集群的统计显著性可以解决。这项资助的研究是在生物学家和计算机科学家的持续合作的背景下进行的。

An important and visible trend in empirical science today isthe increasing prevalence of large data sets that contain from thousands tohundreds of millions of measurements. Examples include data sets arisingfrom high throughput measurement techniques such as gene expression arrays,proteomics and computer network monitoring. While the analysis of largedata sets is important to scientists, it is often outside the realm ofclassical statistical methods, and frequently presents new conceptual andcomputational challenges. The funded research has two principle parts. Inthe first, the investigators are studying the application of a relativelynew development in the field of Data Mining, known as subspace clustering,to the exploratory statistical analysis of high dimensional data. In thesecond, the investigators are applying ideas from Statistics and Probabilityto the development of new subspace clustering methods, and to rigorousmathematical analyses of their results. Research is being carried out inthe context of ongoing collaborations with biological scientists, and isbeing incorporated in software that will be used by the collaboratingscientists to identify and assess significant sample-variableassociations in a variety of large data sets.An important and visible trend in empirical science today is the increasing prominence of large data sets that contain fromthousands to hundreds of millions ofmeasurements. Examples include data sets arising from high throughputmeasurement techniques such as gene expression arrays, proteomics andcomputer network monitoring. Whereas small to moderate data sets typicallyhave more samples than measurements, in large data sets it is common to havemore measurements than samples, so-called ``high dimension and low samplesize''. The investigators are studying the application of data miningmethods known as subspace clustering to the exploratory analysis of highdimensional data. Subspace clustering identifies distinguished samplevariable interactions (submatrices) in a given data matrix. Unlike standardtwo-way clustering, the sample and variable sets for different clusters canoverlap. The investigators are investigating the noise sensitivity ofexisting subspace clustering algorithms, and are developing andimplementing new subspace clustering methods for average based selectioncriteria that are better suited for applications where noise is present.As an application of these methods, they are using subspace clusters toclassify high dimensional data. Using a variety of tools from combinatorialprobability, the investigators are also developing a rigorous theoreticalframework in which multiple testing and the statistical significance ofsubspace clusters can be addressed. The funded research is being carried outin the context of ongoing collaborations with biologists and computerscientists.