Collaborative Research: Efficient Parallel Iterative Monte Carlo Methods for Statistical Analysis of Big Data

计算机技术与科学和日常生活的融合使大量数据的收集成为可能。为了分析这些数据，人们可能不得不求助于并行和分布式架构。虽然并行和分布式架构为大数据的存储和操作提供了新的能力，但从推理的角度来看，目前的统计方法如何被转移到大数据的范式中还不清楚。此外，数据大小的增长通常伴随着数据结构和解释结构所需的模型的复杂性的增长。虽然迭代蒙特卡罗算法，如马尔可夫链蒙特卡罗（MCMC）、随机逼近和期望最大化（EM）算法，已经被证明是非常强大和典型的独特的复杂结构数据分析计算工具，但它们对于大数据来说是不可行的，因为大数据通常需要大量迭代和每次迭代都需要完整扫描整个数据集。大数据对当前的统计方法提出了巨大的挑战。研究人员提出了开发蒙特卡罗算法的一般原则，该算法适用于大数据，并适用于并行和分布式架构；也就是说，使用从子样本并行计算的蒙特卡洛平均值来近似最初需要从完整数据集计算的数量。该原则避免了在算法迭代中重复扫描完整数据的要求，同时使算法能够对所考虑的问题产生统计上合理的解决方案。根据这一原理，提出了一种通用算法，即基于次抽样近似的并行随机近似算法，用于大数据问题的参数估计。与现有的算法不同，如小自举包、聚合估计方程和分而治之算法，所提出的算法适用于观测值通常依赖的问题。基于同样的原理，提出了一种基于次抽样近似的并行Metropolis-Hastings算法用于大数据贝叶斯分析，并提出了一种基于次抽样近似的并行Monte Carlo EM算法用于缺失观测值的大数据问题的参数估计。除了基于子采样近似的并行迭代蒙特卡罗算法外，基于流行的分治思想，提出了一种用于大数据贝叶斯分析的尴尬并行MCMC算法。提出了不同的数据集划分和结果聚合方案。本文将严格研究所提出的并行迭代蒙特卡罗算法的有效性，包括基于次抽样近似和尴尬并行的算法。该算法将应用于卫星气候数据的时空建模、全基因组关联研究和流数据分析。这个项目的智力价值在于提出了大数据统计分析的一般原则：使用子样本的蒙特卡罗平均值来近似最初需要从完整数据集计算的数量。这一原则为将当前的统计方法转换为大数据范式提供了一个总体策略。在此原理下，提出了几种基于子采样近似的并行迭代蒙特卡罗算法。提出的算法解决了大数据分析的核心问题：如何在避免对整个数据集重复扫描的同时，对大数据进行统计上合理的分析？这个项目将产生更广泛的影响，因为大数据在几乎所有的科技领域都无处不在。一个成功的并行迭代蒙特卡罗计算理论和方法研究项目可以在整个科学技术领域产生巨大的效益。研究成果将通过与这些学科的研究人员的直接合作、会议报告、书籍和将在学术期刊上发表的论文，传播给感兴趣的社区，例如大气科学、生物医学科学、工程和社会科学。该项目还将通过研究生直接参与项目并将结果纳入本科和研究生课程，对教育产生重大影响。此外，将在该项目下开发的分布式迭代统计计算包（DISC）旨在为博士生和研究人员提供一个平台，这些研究人员拥有网络连接的计算机，可以在并行或更准确地说，网格计算环境中实验开发高效迭代蒙特卡罗算法的新想法。

The integration of computer technology into science and daily life has enabled the collection of massive volumes of data. To analyze these data, one may have to resort to parallel and distributed architectures. While the parallel and distributed architectures present new capabilities for storage and manipulation of big data, it is unclear, from the inferential point of view, how the current statistical methodology can be transported to the paradigm of big data. Also, growing data size typically comes together with a growing complexity of data structures and of the models needed to account for the structures. Although iterative Monte Carlo algorithms, such as the Markov chain Monte Carlo (MCMC), stochastic approximation, and expectation-maximization (EM) algorithms, have proven to be very powerful and typically unique computational tools for analyzing data of complex structures, they are infeasible for big data as for which a large number of iterations and a complete scan of the full dataset for each iteration are typically required. Big data have put a great challenge on the current statistical methodology. The investigators propose a general principle for developing Monte Carlo algorithms that are feasible for big data and workable on parallel and distributed architectures; that is, using Monte Carlo averages calculated in parallel from subsamples to approximate the quantities that originally need to calculate from the full dataset. This principle avoids the requirement for repeated scans of full data in algorithm iterations, while enabling the algorithm to produce statistically sensible solutions to the problem under consideration. Under this principle, a general algorithm, the so-called subsampling approximation-based parallel stochastic approximation algorithm, is proposed for parameter estimation for big data problems. Unlike the existing algorithms, such as the bag of little bootstraps, aggregated estimation equation, and split-and-conquer algorithms, the proposed algorithm works for the problems for which the observations are generally dependent. Under the same principle, a subsampling approximation-based parallel Metropolis-Hastings algorithm is proposed for Bayesian analysis of big data, and a subsampling approximation-based parallel Monte Carlo EM algorithm is proposed for parameter estimation for the big data problems with missing observations. In addition to the subsampling approximation-based parallel iterative Monte Carlo algorithms, an embarrassingly parallel MCMC algorithm is proposed for Bayesian analysis of big data based on the popular idea of divide-and-conquer. Various schemes of dataset partition and results aggregation are proposed. The validity of the proposed parallel iterative Monte Carlo algorithms, including both the subsampling approximation-based and embarrassingly parallel ones, will be rigorously studied. The proposed algorithms will be applied to spatio-temporal modeling of satellite climate data, genome-wide association study, and stream data analysis.The intellectual merit of this project is to propose a general principle for statistical analysis of big data: Using Monte Carlo averages of subsamples to approximate the quantities that originally need to calculate from the full dataset. This principle provides a general strategy for transporting the current statistical methodology to the paradigm of big data. Under this principle, a few subsampling approximation-based parallel iterative Monte Carlo algorithms are proposed. The proposed algorithms address the core problem of big data analysis:how to make a statistically sensible analysis for big data while avoiding repeated scans of the full dataset? This project will have broader impacts because big data are ubiquitous throughout almost all fields of science and technology. A successful research program in theory and methods of parallel iterative Monte Carlo computations can have immense benefit widely throughout science and technology. The research results will be disseminated to the communities of interest, such as atmospheric science, biomedical science, engineering, and social science, via direct collaboration with researchers in these disciplines, conference presentations, books, and papers to be published in academic journals. The project will have also significant impacts on education through direct involvement of graduate students in the project and incorporation of results into undergraduate and graduate courses. In addition, the package Distributed Iterative Statistical Computing (DISC) that will be developed under this project is designed to provide a platform for Ph.D. students and researchers like the investigators with network-connected computers to experiment new ideas of developing efficient iterative Monte Carlo algorithms in parallel or, more exactly, grid computing environments.