Collaborative Research: Scalable Bayesian Methods for Complex Data with Optimality Guarantees

数据获取、处理和存储方面的惊人进步为分析各种应用中不断增长的大小和复杂性的数据集提供了机会，如社会和生物网络、流行病学、基因组学和互联网推荐系统。在这些数据的巨大规模和维度之下，往往存在着一种简约的结构。在这方面，统计推断的贝叶斯方法很有吸引力，因为它通过先验分布纳入结构假设，能够对复杂现象进行概率建模，并提供不确定性的自动表征。该研究项目旨在促进获取和转化关于大数据低维骨架的先验知识，以在保持计算效率的同时提供现实的不确定性表征。贝叶斯计算在高维和大数据问题中提出了巨大的挑战。这项研究的目的是开发尖端的计算策略和软件包，以供公开实施。该项目吸引了研究生参与研究。研究项目重点是贝叶斯方法在高维和大数据问题中的理论基础和计算策略，这些问题的动机是在社会网络和流行病学中的应用。系统地开发和评估高维问题中的先验分布的技术将被研究，特别强调在统计效率和计算可伸缩性之间的权衡。具体方向包括具有收缩先验的后验抽样的高效算法、大数据问题中分而治之策略的理论框架、在具有未知社区数量的大型网络中对节点进行聚类的快速算法，以及在稀疏列联表中发现结构的方法。这些算法将受到对后验分布行为的严格理论理解的推动，特别强调分布式计算框架中的不确定性的适当量化。将为每个应用程序开发软件。

Spectacular advances in data acquisition, processing, and storage present the opportunity to analyze datasets of ever-increasing size and complexity in various applications, such as social and biological networks, epidemiology, genomics, and Internet recommender systems. Underlying the massive size and dimension of these data, there is often a parsimonious structure. The Bayesian approach to statistical inference is attractive in this context in terms of incorporating structural assumptions through prior distributions, enabling probabilistic modeling of complex phenomenon, and providing an automatic characterization of uncertainty. This research project aims to advance eliciting and translating prior knowledge regarding the low-dimensional skeleton of big data to provide realistic uncertainty characterizations while maintaining computational efficiency. Bayesian computation poses substantial challenge in high-dimensional and big data problems. The research aims to develop cutting-edge computational strategies and software packages for implementation to be made available publicly. The project involves graduate students in the research.The research project focuses on theoretical foundations and computational strategies for Bayesian methods in high-dimensional and big data problems motivated by applications in social networks and epidemiology. Techniques for systematically developing and evaluating prior distributions in high-dimensional problems will be investigated with a special emphasis on the trade-off between statistical efficiency and computational scalability. Specific directions include efficient algorithms for posterior sampling with shrinkage priors, a theoretical framework for divide and conquer strategies in big data problems, fast algorithms for clustering nodes in large networks with unknown number of communities, and methods for discovering structure in sparse contingency tables. The algorithms will be motivated by rigorous theoretical understanding of the behavior of the posterior distribution with a particular emphasis on proper quantification of uncertainty in a distributed computing framework. Software will be developed for each application.