General Purpose Methods for Unsupervised Exploration of Large Datasets

点击翻译按钮获取中文摘要

With the enormous stream of electronic data being continually generated,it is imperative to use unsupervised methods to organize and explorethe data. The enormous quantity of data precludes the use of supervisedmethods. The goal of this project is to develop unsupervised methodscapable of organizing and annotating large datasets of unknown structure,facilitating further exploration and analysis of the data.With the recent advent of general purpose, scalable, unsupervisedclustering methods such as Principal Direction Divisive Partitioning(PDDP), whole new vistas open up in the uses and applications ofunsupervised methods. This particular method yields additionalby-products for free: a hierarchical structure, and identification ofthe most distinctive attributes. These by-products are just the itemsneeded to impose a structure on a dataset and annotate the computedstructure at various levels of detail.This naturally leads to this project: to develop scalable generalpurpose clustering methods, to extract the information needed toannotate the datasets so users can effectively navigate through thedata, and to perform the associated statistical and theoretical analyses.The methods will be validated on a wide variety of domains, including theWWW, specialized legal and/or medical databases, astronomical catalogs,and genomics.