High-dimensional Clustering: Theory and Methods

过去20年见证了科学和工程中出现的数据集的规模和复杂性的爆炸性增长。广泛地说，发现数据中潜在结构的聚类方法是我们导航、探索和可视化海量数据集的主要工具。这些方法在系统发育学、医学、精神病学、考古人类学、植物社会学、经济学等多个领域得到了广泛而成功的应用。尽管它无处不在，但由于缺乏针对高维数据集的灵活的聚类方法，以及在聚类问题中缺乏有意义的推理保证，聚类方法的广泛科学采用受到了阻碍。因此，这项研究的目标是开发新的和有效的方法来对复杂的数据集进行聚类，并进一步为这些方法建立推理基础--这将反过来导致可操作的结论。这项研究将导致新的聚类方法的发展，以及对旨在揭示数据中潜在结构的方法的根本局限性的更深入的理解。本项目的研究构成部分由四个目标组成，旨在解决这一高级别目标的相关方面：(A)分析和开发高维数据集的新的聚类方法，特别侧重于基于混合模型的聚类和最小体积聚类等实用方法；(B)受科学应用的推动，开发在聚类的背景下进行推理的新方法，在科学应用中，不仅重要的是对数据进行聚类，而且要清楚地描述所发现的簇的抽样变异性；(C)制定高维数据集的基本下限；(D)制定具有推理保证的功能数据的新的聚类方法。这些研究构成部分与具体的教育举措密切结合在一起，包括开发和广泛传播公开提供的高维分类软件；在机器学习会议上提供教程和讲习班，并促进统计部门和卡内基梅隆大学机器学习之间的进一步互动。

The past two decades have witnessed an explosion in the scale and complexity of data sets that arise in science and engineering. Broadly, clustering methods which discover latent structure in data are our primary tool for navigating, exploring and visualizing massive datasets. These methods have been widely and successfully applied in phylogeny, medicine, psychiatry, archaeology and anthropology, phytosociology, economics and several other fields. Despite its ubiquity, the widespread scientific adoption of clustering methods have been hindered by the lack of flexible clustering methods for high-dimensional datasets and by the dearth of meaningful inferential guarantees in clustering problems. Accordingly, the goal of this research is to develop new and effective methods for clustering complex data-sets, and to further develop an inferential grounding -- which will in turn lead to actionable conclusions -- for these methods. This research will lead to the development of new clustering methods, as well as to a deeper understanding of the fundamental limitations of methods aimed at uncovering latent structure in data. The research component of this project consists of four aims designed to address related aspects of this high-level goal: (a) analyze and develop new clustering methods for high-dimensional datasets, with a particular focus on practically useful methods like mixture-model based clustering, and minimum volume clustering; (b) develop novel methods for inference in the context of clustering, motivated by scientific applications where it is important not only to cluster the data but also to clearly characterize the sampling variability of the discovered clusters; (c) develop fundamental lower bounds for high-dimensional clustering (d) develop novel methods for clustering functional data with inferential guarantees. These research components are closely coupled with concrete educational initiatives, including the development and broad dissemination of publicly-available software for high-dimensional clustering; tutorials and workshops at Machine Learning conferences and fostering further interactions between the Departments of Statistics and Machine Learning at Carnegie Mellon.