Analysis of High Dimensional Data Using Subspace Clustering

点击翻译按钮获取中文摘要

An important and visible trend in empirical science today isthe increasing prevalence of large data sets that contain from thousands tohundreds of millions of measurements. Examples include data sets arisingfrom high throughput measurement techniques such as gene expression arrays,proteomics and computer network monitoring. While the analysis of largedata sets is important to scientists, it is often outside the realm ofclassical statistical methods, and frequently presents new conceptual andcomputational challenges. The funded research has two principle parts. Inthe first, the investigators are studying the application of a relativelynew development in the field of Data Mining, known as subspace clustering,to the exploratory statistical analysis of high dimensional data. In thesecond, the investigators are applying ideas from Statistics and Probabilityto the development of new subspace clustering methods, and to rigorousmathematical analyses of their results. Research is being carried out inthe context of ongoing collaborations with biological scientists, and isbeing incorporated in software that will be used by the collaboratingscientists to identify and assess significant sample-variableassociations in a variety of large data sets.An important and visible trend in empirical science today is the increasing prominence of large data sets that contain fromthousands to hundreds of millions ofmeasurements. Examples include data sets arising from high throughputmeasurement techniques such as gene expression arrays, proteomics andcomputer network monitoring. Whereas small to moderate data sets typicallyhave more samples than measurements, in large data sets it is common to havemore measurements than samples, so-called ``high dimension and low samplesize''. The investigators are studying the application of data miningmethods known as subspace clustering to the exploratory analysis of highdimensional data. Subspace clustering identifies distinguished samplevariable interactions (submatrices) in a given data matrix. Unlike standardtwo-way clustering, the sample and variable sets for different clusters canoverlap. The investigators are investigating the noise sensitivity ofexisting subspace clustering algorithms, and are developing andimplementing new subspace clustering methods for average based selectioncriteria that are better suited for applications where noise is present.As an application of these methods, they are using subspace clusters toclassify high dimensional data. Using a variety of tools from combinatorialprobability, the investigators are also developing a rigorous theoreticalframework in which multiple testing and the statistical significance ofsubspace clusters can be addressed. The funded research is being carried outin the context of ongoing collaborations with biologists and computerscientists.