BIGDATA: Collaborative Research: F: Big Data, It's Not So Big: Exploiting Low-Dimensional Geometry for Learning and Inference

这项研究将利用代数和微分几何的思想来解决现代高维和大规模数据科学中的核心问题。该项目将开发统计方法和数值工具，以坚实的数学、统计和计算基础为基础，从大量数据中提取低维几何形状，并应用于聚类、数据汇总、预测、降维和可视化。作为该项目的一部分，开发的解决方案可以在生物学、医学、社会科学、通信网络和工程等不同领域的实际应用中带来根本性的进步。除了通过统计和数学理论和模拟研究进行内部验证外，项目中开发的方法还将涉及通过跨学科应用进行外部验证。这些应用包括：(1)从基因组数据推断种群结构；(2)通过主题模型进行文献分析；(3)推测与黑色素瘤耐药相关的基因网络亚群。这项研究的核心前提是，即使数据量可能是巨大的，一个紧凑的模型可以表示这些数据。具体来说，高维和/或大量数据可以通过子空间的混合来合理地近似，其中存在稀疏表示。潜在不同维度的子空间的混合是一种灵活、丰富的数据表示，具有良好的数学属性，可以扩展到大数据。子空间混合建模的几个基本挑战将在本研究中得到解决：1)子空间将具有不同的维度；2)子空间参数和混合参数都需要推断；3)高维和海量数据都需要有效的推理算法。所有这些挑战的核心基础障碍是模型是一个分层空间（流形的并集），因此具有奇点。本研究的关键观点是，模型空间中存在缓解这些奇异性的嵌入和表示。这些想法被实现为具体的贝叶斯、频率论和数值算法和模型，以解决上面列出的现实世界的例子。

This research will leverage ideas from algebraic and differential geometry to address core problems in modern high-dimensional and massive data science. The project will develop statistical methods and numerical tools, grounded in solid mathematical, statistical, and computational foundations, to extract low dimensional geometry from massive data with applications in clustering, data summarization, prediction, dimension reduction, and visualization. The solutions developed as part of this project can result in fundamental advances in practical applications across fields as diverse as biology, medicine, social sciences, communication networks, and engineering. In addition to internal validation via statistical and mathematical theory and simulation studies, the methods developed in the project will involve external validation via interdisciplinary applications. These applications include: (1) inference of population structure from genomic data; (2) document analysis via topic models; and (3) inference of subsets of putative gene networks relevant to drug resistance in melanoma.The research is motivated by the central premise that, even though the amount of data may be massive, a compact model can represent these data. Specifically, high-dimensional and/or massive data can be reasonably approximated by a mixture of subspaces, for which sparse representations exist. A mixture of subspaces of potentially different dimensions is a flexible, rich representation of data with nice mathematical properties that can scale to large data. There are several fundamental challenges in modeling mixtures of subspaces that will be addressed in this research: 1) the subspaces will be of different dimensions, 2) both the subspace parameters and the mixing parameters need to be inferred, 3) efficient algorithms for inference are required for both high-dimensional and massive data. The central foundational impediment in all of these challenges is that the model is a stratified space (a union of manifolds), and therefore has singularities. The key insight in this research is that there exist embeddings and representations of the model space that mitigate these singularities. These ideas are implemented as concrete Bayesian, frequentist, and numerical algorithms and models to address the real world examples listed above.