BIGDATA: F: Collaborative Research: Moment Methods for Big Data: Modern Theory, Algorithms, and Applications

现代科学学科越来越多地面临着越来越大的规模和复杂性的数据集。实验观测可能会受到不准确的测量和缺失值的影响，而生命科学中现代高通量实验程序的输出量之大，使数据处理成为一个越来越大的挑战。从这些数据中得出准确的科学推论需要开发理论上合理且计算效率高的新工具。该项目旨在开发统计方法，以揭示大型复杂数据的内在结构。计划中的方法有可能成为许多科学和工程学科中使用的默认数据科学技术。将公开提供快速、用户友好的软件，用于通用的大数据分析和具体的科学应用。计划方法的第一个支柱是主成分分析。研究人员正在将PCA的使用扩展到具有损坏观测、非高斯噪声和低信噪比的高维观测的设置。这些类型的数据集出现在诸如冷冻电子显微镜和X射线自由电子激光成像等问题中。这项工作将为这些问题的探索性数据分析提供强大的工具。研究计划的第二个支柱是矩量法，这是一种经典的参数估计技术，研究人员已将其用于新问题。研究人员将把矩量法的适用范围扩展到许多表现出一定代数结构的大数据问题。对于这些问题，矩量法能够实现可扩展的和接近最优的统计推断。最后，PCA和矩量法的新扩展将被结合起来，以获得新的接近最佳和可扩展的统计推断程序的高维problems.This奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

Modern scientific disciplines are increasingly faced with datasets of ever larger size and complexity. Experimental observations may be marred by inaccurate measurements and missing values, and the sheer volume of the output of modern high-throughput experimental procedures in the life sciences makes data processing an increasing challenge. Drawing accurate scientific inferences from such data requires developing new tools that are both theoretically sound and computationally efficient. This project aims to develop statistical methodologies for uncovering the intrinsic structure in large, complex data. The planned methods have the potential to become the default data science techniques used in many scientific and engineering disciplines. Fast, user-friendly software will be made publicly available, both for general purpose big data analysis and specific scientific applications.The first pillar of the planned methodology is principal component analysis (PCA). The investigators are extending the use of PCA to the setting of high-dimensional observations with corrupted observations, non-Gaussian noise, and low signal-to-noise ratios. These kinds of datasets arise in problems such as cryo-electron microscopy and X-ray free electron laser imaging. This work will provide robust tools for exploratory data analysis for these problems. The second pillar of the research program is the method of moments, a classical technique for parameter estimation that the investigators have repurposed for new problems. The investigators will extend the range of applicability of the method of moments to many big data problems that exhibit certain algebraic structure. For these problems, the method of moments enables scalable and near-optimal statistical inference. Finally, the novel extensions of PCA and the method of moments will be combined to derive new near-optimal and scalable statistical inference procedures for high-dimensional problems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.