General Purpose Methods for Unsupervised Exploration of Large Datasets

随着大量电子数据流的不断产生，使用无监督的方法来组织和开发这些数据势在必行。 大量的数据排除了监督方法的使用。 该项目的目标是开发能够组织和注释未知结构的大型数据集的无监督方法，以促进对数据的进一步探索和分析，随着最近通用的、可扩展的、无监督的聚类方法如主方向划分（PDDP）的出现，在无监督方法的使用和应用方面开辟了全新的前景。 这种特殊的方法免费产生了额外的副产品：一个层次结构，以及最独特的属性的识别。 这些副产品只是在数据集上施加结构并在各种细节级别上注释计算结构所需的项目。这自然会导致这个项目：开发可扩展的通用聚类方法，提取注释数据集所需的信息，以便用户可以有效地浏览数据，并进行相关的统计和理论分析。这些方法将在各种领域进行验证，包括万维网，专门的法律的和/或医学数据库，天文目录和基因组学。

With the enormous stream of electronic data being continually generated,it is imperative to use unsupervised methods to organize and explorethe data. The enormous quantity of data precludes the use of supervisedmethods. The goal of this project is to develop unsupervised methodscapable of organizing and annotating large datasets of unknown structure,facilitating further exploration and analysis of the data.With the recent advent of general purpose, scalable, unsupervisedclustering methods such as Principal Direction Divisive Partitioning(PDDP), whole new vistas open up in the uses and applications ofunsupervised methods. This particular method yields additionalby-products for free: a hierarchical structure, and identification ofthe most distinctive attributes. These by-products are just the itemsneeded to impose a structure on a dataset and annotate the computedstructure at various levels of detail.This naturally leads to this project: to develop scalable generalpurpose clustering methods, to extract the information needed toannotate the datasets so users can effectively navigate through thedata, and to perform the associated statistical and theoretical analyses.The methods will be validated on a wide variety of domains, including theWWW, specialized legal and/or medical databases, astronomical catalogs,and genomics.