Adaptive Thresholding for Hierarchical Clustering of Variables, with Connections to Scan Statistics

在具有大型数据集的现代数据分析中，一个共同的目标是检测表现出相似行为的变量组。 这个任务通常被称为集群。 例如，在遗传学和蛋白质组学中，聚类可以揭示科学感兴趣的结构，例如潜在的生物学途径。 除了检测数据中的科学相关结构之外，聚类还可以用于简化数据表示和分析。 最广泛使用的聚类方法之一称为层次聚类。 在层次聚类中，计算每对变量之间的相似性度量，如相关性，然后重复合并相似的变量组。 这就引出了一个基本问题：应该进行多大程度的分组？ 拟议的研究包括几个项目，旨在开发广泛适用的方法，用于确定适当的聚类量，根据数据中存在的相似程度。 由此产生的程序也将提供统计保证的意义上产生groups.This建议的目的是开发实用的程序自适应阈值的层次聚类树状图，当应用到成对的变量相似性。 这些程序将连接到推理保证假聚类错误率的结果聚类。 结果将针对一系列共同的联系和可变的相似性措施。 PI还将在现代遗传学应用中演示这些程序。为了支持这些程序，将开发新的理论来描述变量相似性度量的高阶统计量，包括其联合分布的新渐近界和其最大值的新有限样本界。 这里提出的技术也将应用于其他基于阈值的统计程序，特别是，连接之间提出的工作和自适应阈值程序扫描统计。

In modern data analysis with large data sets, a common goal is to detect groups of variables that exhibit similar behavior. This task is usually referred to as clustering. In genetics and proteomics, for instance, clustering can reveal structures of scientific interest, such as potential biological pathways. On top of detecting scientifically relevant structure in the data, clustering can also be used to simplify data representations and analysis. One of the most widely used approaches to clustering is called hierarchical clustering. In hierarchical clustering, a measure of similarity, like correlation, is computed between each pair of variables, and then similar groups of variables are repeatedly merged. This leads to a fundamental question: how much grouping should be done? The proposed research consists of several projects aimed at developing broadly applicable methods for determining the appropriate amount of clustering, based on the degree of similarity present in the data. The resulting procedures will also provide statistical guarantees on the meaning of the resulting groups.This proposal aims to develop practical procedures for adaptive thresholding of hierarchical clustering dendrograms, when applied to pairwise similarities of variables. These procedures will be connected to inferential guarantees about the false cluster error rate of the resulting clustering. The results will target a range of common linkages and variable similarity measures. The PI will also demonstrate these procedures in a modern genetics application.To support these procedures, new theory will be developed describing the large order statistics of variable similarity measures, including new asymptotic bounds on their joint distributions and new finite-sample bounds on their maxima. The techniques proposed here will also have application to other threshold-based procedures in statistics; in particular, connections may be made between the proposed work and adaptive thresholding procedures for scan statistics.