Collaborative Research: Scalable Bayesian Methods for Complex Data with Optimality Guarantees

数据采集、处理和存储方面的惊人进步为分析各种应用中不断增长的规模和复杂性的数据集提供了机会，例如社会和生物网络、流行病学、基因组学和互联网推荐系统。在这些数据的庞大规模和维度背后，往往存在一个简约的结构。在这种情况下，贝叶斯方法在统计推断方面很有吸引力，因为它通过先验分布结合结构假设，实现复杂现象的概率建模，并提供不确定性的自动表征。本研究项目旨在促进对大数据低维骨架的先验知识的提取和转化，在保持计算效率的同时提供现实的不确定性表征。贝叶斯计算对高维和大数据问题提出了实质性的挑战。这项研究的目的是开发先进的计算策略和软件包，以供公众使用。这个项目有研究生参与研究。该研究项目重点研究贝叶斯方法在高维和大数据问题中的理论基础和计算策略，这些问题是由社会网络和流行病学应用驱动的。系统地开发和评估高维问题中的先验分布的技术将被研究，特别强调统计效率和计算可扩展性之间的权衡。具体方向包括具有收缩先验的后验抽样的高效算法，大数据问题中分而治之策略的理论框架，具有未知数量社区的大型网络中节点聚类的快速算法，以及在稀疏列联表中发现结构的方法。这些算法将受到对后验分布行为的严格理论理解的激励，并特别强调分布式计算框架中不确定性的适当量化。将为每个应用程序开发软件。

Spectacular advances in data acquisition, processing, and storage present the opportunity to analyze datasets of ever-increasing size and complexity in various applications, such as social and biological networks, epidemiology, genomics, and Internet recommender systems. Underlying the massive size and dimension of these data, there is often a parsimonious structure. The Bayesian approach to statistical inference is attractive in this context in terms of incorporating structural assumptions through prior distributions, enabling probabilistic modeling of complex phenomenon, and providing an automatic characterization of uncertainty. This research project aims to advance eliciting and translating prior knowledge regarding the low-dimensional skeleton of big data to provide realistic uncertainty characterizations while maintaining computational efficiency. Bayesian computation poses substantial challenge in high-dimensional and big data problems. The research aims to develop cutting-edge computational strategies and software packages for implementation to be made available publicly. The project involves graduate students in the research.The research project focuses on theoretical foundations and computational strategies for Bayesian methods in high-dimensional and big data problems motivated by applications in social networks and epidemiology. Techniques for systematically developing and evaluating prior distributions in high-dimensional problems will be investigated with a special emphasis on the trade-off between statistical efficiency and computational scalability. Specific directions include efficient algorithms for posterior sampling with shrinkage priors, a theoretical framework for divide and conquer strategies in big data problems, fast algorithms for clustering nodes in large networks with unknown number of communities, and methods for discovering structure in sparse contingency tables. The algorithms will be motivated by rigorous theoretical understanding of the behavior of the posterior distribution with a particular emphasis on proper quantification of uncertainty in a distributed computing framework. Software will be developed for each application.