Collaborative Research: Efficient Parallel Iterative Monte Carlo Methods for Statistical Analysis of Big Data

计算机技术与科学和日常生活的结合使得收集海量数据成为可能。要分析这些数据，可能不得不求助于并行和分布式体系结构。虽然并行和分布式架构为大数据的存储和处理提供了新的能力，但从推理的角度来看，目前的统计方法如何才能移植到大数据范式中尚不清楚。此外，不断增长的数据大小通常伴随着数据结构和考虑这些结构所需的模型的日益复杂。尽管马尔科夫链蒙特卡罗(MCMC)、随机逼近和期望最大化(EM)等迭代蒙特卡罗算法已被证明是用于分析复杂结构的数据的非常强大且通常独特的计算工具，但它们对于大数据是不可行的，因为对于大数据，通常需要大量迭代和每次迭代的完整数据集的完整扫描。大数据对当前的统计方法提出了巨大的挑战。研究人员提出了一个通用原则，用于开发对大数据可行、在并行和分布式架构上可行的蒙特卡罗算法；即，使用从子样本并行计算的蒙特卡罗平均值来近似从整个数据集中最初需要计算的量。这一原则避免了在算法迭代中重复扫描完整数据的要求，同时使算法能够为所考虑的问题产生统计上合理的解决方案。在这一原则下，提出了一种大数据参数估计的通用算法--基于次抽样近似的并行随机逼近算法。不同于现有的算法，如小Bootstraps包、聚合估计方程和分裂-征服算法，该算法适用于通常依赖于观测值的问题。在相同的原理下，提出了大数据贝叶斯分析的基于次抽样近似的并行Metropolis-Hastings算法，以及用于大数据缺失观测问题参数估计的基于次抽样近似的并行蒙特卡罗EM算法。除了基于次抽样近似的并行迭代蒙特卡罗算法外，基于流行的分而治之思想，提出了一种用于大数据贝叶斯分析的令人尴尬的并行MCMC算法。提出了多种数据集划分和结果聚合方案。我们将严格研究所提出的并行迭代蒙特卡罗算法的有效性，包括基于二次抽样近似的算法和尴尬的并行算法。该算法将被应用于卫星气候数据的时空建模、全基因组关联性研究和流数据分析。该项目的智力优势在于提出了大数据统计分析的一般原则：使用子样本的蒙特卡罗平均来逼近原始需要从全数据集中计算的量。这一原则为将当前的统计方法转变为大数据范式提供了一种总体战略。在这一原则下，提出了几种基于次采样近似的并行迭代蒙特卡罗算法。提出的算法解决了大数据分析的核心问题：如何对大数据进行统计上合理的分析，同时避免重复扫描整个数据集？这一项目将产生更广泛的影响，因为大数据几乎无处不在，几乎遍及所有科技领域。一个成功的并行迭代蒙特卡罗计算理论和方法的研究项目可以在整个科学和技术领域产生巨大的好处。研究成果将通过与这些学科的研究人员直接合作、会议报告、书籍和将在学术期刊上发表的论文，向感兴趣的社区传播，如大气科学、生物医学科学、工程和社会科学。该项目还将通过研究生直接参与该项目并将成果纳入本科生和研究生课程，对教育产生重大影响。此外，将在该项目下开发的分布式迭代统计计算(DISC)包旨在为博士生和研究人员(如拥有联网计算机的研究人员)提供一个平台，以试验在并行或更准确地说，在网格计算环境中开发高效迭代蒙特卡罗算法的新想法。

The integration of computer technology into science and daily life has enabled the collection of massive volumes of data. To analyze these data, one may have to resort to parallel and distributed architectures. While the parallel and distributed architectures present new capabilities for storage and manipulation of big data, it is unclear, from the inferential point of view, how the current statistical methodology can be transported to the paradigm of big data. Also, growing data size typically comes together with a growing complexity of data structures and of the models needed to account for the structures. Although iterative Monte Carlo algorithms, such as the Markov chain Monte Carlo (MCMC), stochastic approximation, and expectation-maximization (EM) algorithms, have proven to be very powerful and typically unique computational tools for analyzing data of complex structures, they are infeasible for big data as for which a large number of iterations and a complete scan of the full dataset for each iteration are typically required. Big data have put a great challenge on the current statistical methodology. The investigators propose a general principle for developing Monte Carlo algorithms that are feasible for big data and workable on parallel and distributed architectures; that is, using Monte Carlo averages calculated in parallel from subsamples to approximate the quantities that originally need to calculate from the full dataset. This principle avoids the requirement for repeated scans of full data in algorithm iterations, while enabling the algorithm to produce statistically sensible solutions to the problem under consideration. Under this principle, a general algorithm, the so-called subsampling approximation-based parallel stochastic approximation algorithm, is proposed for parameter estimation for big data problems. Unlike the existing algorithms, such as the bag of little bootstraps, aggregated estimation equation, and split-and-conquer algorithms, the proposed algorithm works for the problems for which the observations are generally dependent. Under the same principle, a subsampling approximation-based parallel Metropolis-Hastings algorithm is proposed for Bayesian analysis of big data, and a subsampling approximation-based parallel Monte Carlo EM algorithm is proposed for parameter estimation for the big data problems with missing observations. In addition to the subsampling approximation-based parallel iterative Monte Carlo algorithms, an embarrassingly parallel MCMC algorithm is proposed for Bayesian analysis of big data based on the popular idea of divide-and-conquer. Various schemes of dataset partition and results aggregation are proposed. The validity of the proposed parallel iterative Monte Carlo algorithms, including both the subsampling approximation-based and embarrassingly parallel ones, will be rigorously studied. The proposed algorithms will be applied to spatio-temporal modeling of satellite climate data, genome-wide association study, and stream data analysis.The intellectual merit of this project is to propose a general principle for statistical analysis of big data: Using Monte Carlo averages of subsamples to approximate the quantities that originally need to calculate from the full dataset. This principle provides a general strategy for transporting the current statistical methodology to the paradigm of big data. Under this principle, a few subsampling approximation-based parallel iterative Monte Carlo algorithms are proposed. The proposed algorithms address the core problem of big data analysis:how to make a statistically sensible analysis for big data while avoiding repeated scans of the full dataset? This project will have broader impacts because big data are ubiquitous throughout almost all fields of science and technology. A successful research program in theory and methods of parallel iterative Monte Carlo computations can have immense benefit widely throughout science and technology. The research results will be disseminated to the communities of interest, such as atmospheric science, biomedical science, engineering, and social science, via direct collaboration with researchers in these disciplines, conference presentations, books, and papers to be published in academic journals. The project will have also significant impacts on education through direct involvement of graduate students in the project and incorporation of results into undergraduate and graduate courses. In addition, the package Distributed Iterative Statistical Computing (DISC) that will be developed under this project is designed to provide a platform for Ph.D. students and researchers like the investigators with network-connected computers to experiment new ideas of developing efficient iterative Monte Carlo algorithms in parallel or, more exactly, grid computing environments.