Distance-Based Analysis for Complex High-Dimensional Data

在整个20世纪，距离在统计的重要领域发挥了重要作用，包括分类、聚类、判别分析、多维尺度、抽样、空间统计、评分规则和机器学习中的核方法。距离对于散度度量、相对熵和信息增益的定义也是至关重要的，其中一些是C.R. Rao的二次熵概念和生态学和其他科学领域的多样性分析的基础。然而，目前在我们的知识和新出现的统计文献中，对复杂的高维数据使用基于距离的测试和分析存在重大差距。一个例子是相似性分析，这是引用最多和使用最广泛的基于距离的统计方法之一，但由于缺乏相关的数学知识而受到限制。这项研究将从各种基于距离的统计方法中获得新的数学知识，并应用这些知识来回答林业、生态学和海洋科学中许多学科中出现的重要科学问题，例如：(1)热带森林的生物多样性如何变化？(2)不同海洋区域细菌群落的分类和功能特征如何随环境条件而变化？该项目建立了多个学科之间和两个美国学术机构之间的合作，并为研究生和本科生提供研究和培训。该项目开发了基于距离的统计方法和计算的新知识体系，用于分析以组合、树、图或网络形式出现的复杂、高维数据。这里考虑的距离都是非欧几里得的——要么是不满足任何三角不等式的非度量差异，要么只是离散数——但它们都是由有条件的正定核产生的。距离的例子包括欧几里得距离的平方，Bray-Curtis不相似度，Jensen-Shannon距离，Unifrac或Kantorovich-Rubinstein度量，艾奇逊距离，编辑距离，各种图核和谱距离，以及基于最优传输问题的其他距离。具体地说，该项目通过沿着数据点的汇集样本的最短哈密顿路径（或循环）计数运行，提出了精确无分布的两样本和多样本运行测试、更改点和其他相关问题的数学理论和计算。该项目还考虑相似性分析和相关的基于距离的等级测试，并得出新的数学结果，使我们能够进行更高级的统计分析。该项目有助于：(i)深入分析热带森林的生物多样性；（ii）研究不同海洋地区原核生物群落的分类和功能特征随环境条件的变化；（三）研究太平洋深海淤泥中稀土元素组成的变异性；（iv）了解俄勒冈海岸潮间带群落在上升流和养分输送方面的关系。该项目集数学研究、科学和教育于一体，并将为研究生提供论文工作的机会。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Throughout the course of the twentieth century, distances have played a significant role in important areas of statistics, which include classification, clustering, discriminant analysis, multidimensional scaling, sampling, spatial statistics, scoring rules, and kernel methods in machine learning. Distances are also central to the definition of divergence measures, relative entropy and information gain, some of which are fundamental to the concept of C.R. Rao's quadratic entropy and the analysis of diversity in ecology and other areas of science. Yet, at present there are significant gaps in our knowledge and in the emerging statistical literature on the use of distance-based tests and analyses for complex high dimensional data. One example is analysis of similarity which is among the most cited and most widely used distance-based statistical methods but is limited by an absence of relevant mathematical knowledge. This research will derive new mathematical knowledge on various distance-based statistical methods, and apply this for providing answers to important scientific questions arising in a number of disciplines in forestry, ecology and marine science, such as: (1) how biodiversity changes in tropical forests? (2) how taxonomic and functional profiles of bacterial communities change with environmental conditions in different oceanic regions? The project establishes collaborations among several disciplines and between two US academic institutions and provides research and training to graduate and undergraduate students. The project develops a new body of knowledge on distance-based statistical methods and computation for analyzing complex, high dimensional data that arise in the form of compositions, trees, graphs, or networks. The distances considered here are all non-Euclidean -- either non-metric dissimilarities that do not satisfy any triangular inequalities or just discrete numbers -- but they all arise from conditionally positive definite kernels. Examples of distances include the squared Euclidean distance, the Bray-Curtis dissimilarity, the Jensen-Shannon distance, Unifrac or the Kantorovich-Rubinstein metric, the Aitchison distance, the edit distance, various graph kernel and spectral distances, and other distances based on optimal transport problems. Specifically, the project advances the mathematical theory and computation of exact distribution-free two and multi-sample runs tests, change points, and other related problems by counting runs along the shortest Hamiltonian path (or loop) of the pooled sample of data points. The project also considers analysis of similarity and related distance-based rank tests and derives new mathematical results that allow us to pursue more advanced statistical analyses. The project contributes to: (i) a deeper analysis of biodiversity in tropical forest; (ii) an investigation of how taxonomic and functional profiles of prokaryotic communities change with environmental conditions in different oceanic regions; (iii) a study of the variability of composition of rare earth elements in deep-sea muds of the Pacific Ocean; and (iv) an understanding of the relationship of intertidal communities in the Oregon coast with respect to upwelling and nutrient delivery. The project integrates mathematics research, science and education and will provide opportunities for dissertation work for graduate students.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.