Statistical and Computational Tools for Analyzing High-Dimensional Heterogeneous Data

现代技术产生了海量的各种形式的数据。高通量数据不可避免地伴随着极大的异构性和巨大的噪声。例如，一项大规模的基因研究通常涉及具有各种不同属性的人；一个社会网络通常由多个隐藏的社区组成，与外部社区相比，这些社区的内部联系更紧密。虽然原始特征具有较高的环境维度(例如，数千个基因)，但内在结构往往表现出较低的复杂性(例如，个人地理位置的纬度和经度)。对潜在结构的精确提取为解决下游任务铺平了道路。面对统计和计算方面的重大挑战，该项目旨在开发有效的方法，从不同种类的数据中估计和推断潜在结构。这个项目将产生用于科学研究的尖端工具，易于实施的开源软件，以及用于理论分析的新的数学定理。该项目还将为统计教育和研究培训提供大量机会。在第一部分中，目标是开发一种新的灵活的方法来聚类高维数据。这一部分针对的是能够识别非球形甚至非凸聚类的新算法。对混合模型的深入分析带来了理论上的见解，包括严格的有限样本统计误差界和计算的有限迭代收敛保证。在第二部分中，目标是研究以成对关系编码单个对象的信息的异质关系数据。这一部分提供了在现实场景中估计和测试潜在结构的可靠方法，在现实场景中，部分观测数据可能不是随机抽样的。最后，第三部分重点对高维个人属性的社会网络等多个相关数据集进行联合分析。在该项目的前两部分中开发的工具将构成解决最后这一推力的研究目标的基本构件。这一研究结果将为提高统计准确性提供新的高效数据整合策略。拟议的研究倡议包括以公开软件的形式传播新方法和算法，以及关于加强跨学科研究培训和增强统计科学多样性的积极议程。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Modern technologies generate tremendous volumes of data in diverse forms. The high throughput data come inevitably with great heterogeneity and enormous amount of noise. For instance, a large-scale genetic study typically involves people with various different attributes; a social network usually consists of multiple hidden communities with denser internal connections compared to external ones. While the raw features have high ambient dimension (for example, thousands of genes), oftentimes the intrinsic structures exhibit low complexity (for example, latitude and longitude of an individual’s geographic location). Precise extraction of the latent structure paves the way for solving downstream tasks. Faced with the significant challenges in statistics and computation, this project aims to develop efficient methodologies for estimating and inferring latent structures from heterogeneous data. This project will yield cutting-edge tools for scientific study, open-source software for easy implementation, and new mathematical theorems for theoretical analysis. The project will also provide numerous opportunities for statistical education and research training.The project is structured into three parts. In the first part, the goal is to develop a new flexible methodology for clustering high-dimensional data. This part aims at new algorithms that can identify non-spherical and even non-convex clusters. An in-depth analysis of mixture models brings theoretical insights including tight finite-sample statistical error bounds and finite-iteration convergence guarantees for computation. In the second part, the goal is to study heterogeneous relational data that encode the information of individual objects in their pairwise relations. This part yields reliable methods for estimating and testing latent structures in the realistic scenario where the partially observed data may not be uniformly sampled at random. Finally, the third part focuses on the joint analysis of multiple related datasets, such as social networks with high-dimensional personal attributes. The tools developed in the first two parts of the project will constitute fundamental building blocks to address the research goals of this last thrust. The research finding will provide novel efficient data integration strategies for enhanced statistical accuracy. The proposed research initiatives include dissemination of the new methods and algorithms in a form of publicly available software and an active agenda on enhancing interdisciplinary research training and enhancing diversity in statistical sciences.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.