CIF: Small: Cluster Analysis for Highly-correlated, Heavy-tailed, and Higher-order Data

现代科学技术的迅速发展导致了空前规模和复杂性的数据集的产生。数据集中常见的复杂性来源是亚种群的存在。例如，一种疾病可能有几个亚型；顾客可能会被同一产品的不同特点所吸引。聚类分析是一种流行的识别亚种群的工具，它可以对每个亚种群进行精细的调查。该项目开发了新的聚类方法来揭示当代数据集中日益复杂的模式。除了受试者的分配，本研究中的聚类方法还进一步发现了每个亚群的定义特征。研究小组将把这些方法应用于各种现实世界的问题，这些问题有可能影响依赖这些数据集的多个领域。还将提供开源和用户友好的软件。此外，该项目将与教育和外联活动结合起来，包括开设新课程、跨学科培训和指导数学和统计科学领域代表性不足的学生群体。当数据高度相关、重尾和/或包含高阶张量时，经典聚类方法往往效率低下和/或不准确。为了解决高维无监督学习问题中的这些挑战，研究人员追求新的概率模型和统计方法，用于大型复杂数据的聚类。研究者通过变量选择的稀疏性原则和线性投影的降维原则的综合来促进模型的简约性。相应的概率框架能够同时进行变量选择/降维、参数估计和预测。通过对数据集中的噪声进行分离和排除，大大提高了估计和预测的效率。同时，模型中的简约性带来了可扩展的算法和新的统计见解。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Rapidly advances in modern science and technology are resulting in the generation of data sets of unprecedented sizes and complexity. A common source of complexity in data sets is the presence of subpopulations. For example, a disease may have several subtypes; and customers may be attracted by different features of the same product. Cluster analysis is a popular tool to identify subpopulations, which affords a refined investigation on each of them. This project develops novel clustering methods to reveal the increasingly complex patterns within contemporary data sets. In addition to the allocation of subjects, the clustering methods in this research further find the defining features of each subpopulation. The research team will apply these methods to various real-world problems with potential to affect multiple fields that rely on such data sets. Open source and user-friendly software will also be provided. Moreover, this project will be integrated with educational and outreach activities, including new courses, interdisciplinary training, and mentoring of underrepresented student groups in mathematical and statistical sciences. Classical clustering methods tend to be inefficient and/or inaccurate when data are highly correlated, heavy-tailed, and/or comprise higher-order tensors. To address these challenges in high-dimensional unsupervised learning problems, the investigators pursue new probabilistic models and statistical methods for clustering of large and complex data. The investigators promote parsimony in the models by the synthesis of the sparsity principle through variable selection and the dimension reduction principle through linear projections. The pursuant probabilistic frameworks enable simultaneous variable selection/dimension reduction, parameter estimation and prediction. By separating and excluding the noise in the data set, efficiency in estimation and prediction is greatly enhanced. Concurrently, parsimony in the models leads to scalable algorithms and new statistical insights.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.