General Purpose Methods for Unsupervised Exploration of Large Datasets

随着海量电子数据流的不断产生，使用无人监督的方法来组织和开发这些数据势在必行。大量的数据排除了使用监督方法的可能性。该项目的目标是开发能够组织和标注未知结构的大数据集的非监督方法，便于进一步探索和分析数据。随着最近出现的通用、可扩展的无监督聚类方法，如主方向分裂划分(PDDP)，非监督方法的使用和应用开辟了全新的前景。这种特殊的方法免费产生额外的副产品：层级结构和识别最独特的属性。这些副产品只是在数据集上施加结构并在不同细节级别注释计算结构所需的项。这自然导致了这个项目：开发可扩展的通用聚类方法，提取注释数据集所需的信息，以便用户可以有效地浏览数据，并执行相关的统计和理论分析。这些方法将在广泛的领域得到验证，包括WWW、专业法律和/或医学数据库、天文星表和基因组学。

With the enormous stream of electronic data being continually generated,it is imperative to use unsupervised methods to organize and explorethe data. The enormous quantity of data precludes the use of supervisedmethods. The goal of this project is to develop unsupervised methodscapable of organizing and annotating large datasets of unknown structure,facilitating further exploration and analysis of the data.With the recent advent of general purpose, scalable, unsupervisedclustering methods such as Principal Direction Divisive Partitioning(PDDP), whole new vistas open up in the uses and applications ofunsupervised methods. This particular method yields additionalby-products for free: a hierarchical structure, and identification ofthe most distinctive attributes. These by-products are just the itemsneeded to impose a structure on a dataset and annotate the computedstructure at various levels of detail.This naturally leads to this project: to develop scalable generalpurpose clustering methods, to extract the information needed toannotate the datasets so users can effectively navigate through thedata, and to perform the associated statistical and theoretical analyses.The methods will be validated on a wide variety of domains, including theWWW, specialized legal and/or medical databases, astronomical catalogs,and genomics.