BIGDATA: F: Collaborative Research: Moment Methods for Big Data: Modern Theory, Algorithms, and Applications

现代科学学科越来越多地面临着越来越大的规模和复杂性的数据集。实验观察可能会受到不准确的测量和缺失值的损害，并且生命科学中现代高通量实验程序输出的绝对数量使数据处理成为越来越大的挑战。要从这些数据中得出准确的科学推论，就需要开发理论上合理、计算效率高的新工具。该项目旨在开发统计方法，以揭示大型复杂数据的内在结构。计划的方法有可能成为许多科学和工程学科中使用的默认数据科学技术。快速、用户友好的软件将向公众开放，用于一般用途的大数据分析和特定的科学应用。计划方法的第一个支柱是主成分分析（PCA）。研究人员正在将PCA的使用扩展到具有损坏观测，非高斯噪声和低信噪比的高维观测的设置。这些类型的数据集出现在冷冻电子显微镜和x射线自由电子激光成像等问题中。这项工作将为这些问题的探索性数据分析提供强大的工具。研究计划的第二个支柱是矩量法，这是一种用于参数估计的经典技术，研究人员已将其用于解决新问题。研究者将矩量方法的适用范围扩展到许多具有一定代数结构的大数据问题。对于这些问题，矩量方法可以实现可伸缩和接近最优的统计推断。最后，将PCA的新扩展和矩量方法相结合，推导出新的高维问题的近最优和可扩展的统计推理程序。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Modern scientific disciplines are increasingly faced with datasets of ever larger size and complexity. Experimental observations may be marred by inaccurate measurements and missing values, and the sheer volume of the output of modern high-throughput experimental procedures in the life sciences makes data processing an increasing challenge. Drawing accurate scientific inferences from such data requires developing new tools that are both theoretically sound and computationally efficient. This project aims to develop statistical methodologies for uncovering the intrinsic structure in large, complex data. The planned methods have the potential to become the default data science techniques used in many scientific and engineering disciplines. Fast, user-friendly software will be made publicly available, both for general purpose big data analysis and specific scientific applications.The first pillar of the planned methodology is principal component analysis (PCA). The investigators are extending the use of PCA to the setting of high-dimensional observations with corrupted observations, non-Gaussian noise, and low signal-to-noise ratios. These kinds of datasets arise in problems such as cryo-electron microscopy and X-ray free electron laser imaging. This work will provide robust tools for exploratory data analysis for these problems. The second pillar of the research program is the method of moments, a classical technique for parameter estimation that the investigators have repurposed for new problems. The investigators will extend the range of applicability of the method of moments to many big data problems that exhibit certain algebraic structure. For these problems, the method of moments enables scalable and near-optimal statistical inference. Finally, the novel extensions of PCA and the method of moments will be combined to derive new near-optimal and scalable statistical inference procedures for high-dimensional problems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.