BIGDATA: Collaborative Research: F: Big Data, It's Not So Big: Exploiting Low-Dimensional Geometry for Learning and Inference

这项研究将利用代数和微分几何的思想来解决现代高维和大规模数据科学中的核心问题。 该项目将开发统计方法和数值工具，以坚实的数学，统计和计算基础为基础，从海量数据中提取低维几何形状，并应用于聚类，数据汇总，预测，降维和可视化。 作为该项目的一部分开发的解决方案可以在生物学，医学，社会科学，通信网络和工程等不同领域的实际应用中取得根本性的进步。 除了通过统计和数学理论和模拟研究进行内部验证外，该项目中开发的方法还将通过跨学科应用进行外部验证。 这些应用包括：（1）从基因组数据推断群体结构;（2）通过主题模型进行文档分析;（3）推断与黑色素瘤耐药性相关的推定基因网络子集。该研究的中心前提是，即使数据量可能很大，但紧凑的模型可以表示这些数据。 具体而言，高维和/或海量数据可以通过存在稀疏表示的子空间的混合来合理地近似。 潜在不同维度的子空间的混合是一种灵活、丰富的数据表示，具有良好的数学属性，可以扩展到大数据。 在对混合子空间进行建模时，存在几个基本挑战，这些挑战将在本研究中解决：1）子空间将具有不同的维度，2）子空间参数和混合参数都需要推断，3）对于高维和海量数据都需要有效的推断算法。 所有这些挑战的核心基础障碍是模型是一个分层空间（流形的并集），因此具有奇点。 本研究的关键见解是，存在模型空间的嵌入和表示，以减轻这些奇异性。 这些想法被实现为具体的贝叶斯，频率论，和数值算法和模型，以解决上面列出的真实的世界的例子。

This research will leverage ideas from algebraic and differential geometry to address core problems in modern high-dimensional and massive data science. The project will develop statistical methods and numerical tools, grounded in solid mathematical, statistical, and computational foundations, to extract low dimensional geometry from massive data with applications in clustering, data summarization, prediction, dimension reduction, and visualization. The solutions developed as part of this project can result in fundamental advances in practical applications across fields as diverse as biology, medicine, social sciences, communication networks, and engineering. In addition to internal validation via statistical and mathematical theory and simulation studies, the methods developed in the project will involve external validation via interdisciplinary applications. These applications include: (1) inference of population structure from genomic data; (2) document analysis via topic models; and (3) inference of subsets of putative gene networks relevant to drug resistance in melanoma.The research is motivated by the central premise that, even though the amount of data may be massive, a compact model can represent these data. Specifically, high-dimensional and/or massive data can be reasonably approximated by a mixture of subspaces, for which sparse representations exist. A mixture of subspaces of potentially different dimensions is a flexible, rich representation of data with nice mathematical properties that can scale to large data. There are several fundamental challenges in modeling mixtures of subspaces that will be addressed in this research: 1) the subspaces will be of different dimensions, 2) both the subspace parameters and the mixing parameters need to be inferred, 3) efficient algorithms for inference are required for both high-dimensional and massive data. The central foundational impediment in all of these challenges is that the model is a stratified space (a union of manifolds), and therefore has singularities. The key insight in this research is that there exist embeddings and representations of the model space that mitigate these singularities. These ideas are implemented as concrete Bayesian, frequentist, and numerical algorithms and models to address the real world examples listed above.