CAREER: Statistical Inference in High Dimensions using Variational Approximations

现代数据应用通常涉及包含大量观察和特征的大量数据集。为了促进实时统计学习，迫切需要有原则和计算效率高的统计方法。变分推理方法最近在这种情况下成为一种流行的选择。“变分推理”一词指的是为广泛的问题开发统计算法的一种通用的开箱即用策略。例如，这些算法被用作文本挖掘、超逼真人工文本和图像生成、机器翻译等方面的子程序。由于所提出的方法的计算效率和优越的实用性能，这种方法非常有吸引力。尽管有这些优点，但对这些变分方法的严格保证仍处于萌芽状态。这个项目将为这种方法在不同情况下的有效性制定统计保证。随后，这些新的见解将被开发用于现代数据应用的新的统计方法。提出的研究结果将允许实践者有信心地部署变分推理方法。此外，这些结果将为统计学家的工具箱增添一套新的原则性的、计算效率高的方法。PI将在整个研究期间及以后将他的研究和教学相结合。特别是，PI将开发新的本科/研究生课程，重点放在变分推理和指导学生（特别是那些来自代表性不足背景的学生），目的是向他们介绍统计和数据科学的机会。拟议的研究和教育活动将扩大STEM的参与范围，并鼓励统计和数据科学领域的职业发展。该项目将研究基于变分近似的统计推断，重点是三个具体的重点：(i)基于回归模型的朴素平均场（NMF）近似的统计推断，（ii）超越回归的朴素平均场近似和（iii）高级平均场近似。在主题(i)下，PI将为高维线性模型开发经验贝叶斯方法，并使用NMF近似比较贝叶斯变量选择算法。主题（ii）将关注隐马尔可夫随机场和贝叶斯神经网络的NMF逼近。最后，主题（iii）将侧重于某些替代的平均场近似。物理学家推测，如果数据点和特征的数量既大又可比较，则NMF近似值不再准确；相反，thoulless - anderson - palmer （TAP）近似，一种先进的平均场近似，应该有助于贝叶斯最优推断。提出的研究将在比例渐近制度下的高维线性回归的背景下建立这一猜想。所提出的方法的理论基础将基于源自非线性大偏差（在概率和组合学中研究）、自旋玻璃（在概率和统计物理学中研究）和图形模型的不同思想。反过来，这些思想将与经典的统计思想（例如，非参数最大似然）相结合，以开发计算效率高的高维推理方法。这种思想的交叉授粉将在每个领域产生独立的后续研究方向。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Modern data applications routinely involve massive datasets comprising a multitude of observations and features. To facilitate statistical learning in real time, there is an urgent need for principled and computationally efficient statistical methodology. Variational Inference methods have recently emerged as a popular choice in this context. The term "Variational Inference" refers to a general out-of-the-box strategy to develop statistical algorithms for a wide class of problems. For example, these algorithms are used as a sub-routine in text mining, generation of hyper-realistic artificial text and images, machine translation, etc. This approach is extremely attractive due to the computational efficiency of the proposed methods, and their superior practical performance. Despite these advantages, rigorous guarantees for these variational methods are still in a nascent state. This project will develop statistical guarantees for the validity of this approach in diverse settings. Subsequently, these new insights will be exploited to develop novel statistical methodology for modern data applications. The outcome of the proposed research will allow practitioners to deploy Variational Inference methods with confidence. In addition, the outcomes will add a new set of principled, computationally efficient methods to the statistician's toolkit. The PI will interweave his research and teaching throughout the research period and beyond. In particular, the PI will develop new undergraduate/graduate courses focusing on Variational Inference and mentor students (particularly those from under-represented backgrounds) with the aim of introducing them to opportunities in statistics and data science. The proposed research and educational activities will broaden participation in STEM generally, and encourage careers in statistics and data science.This project will study statistical inference based on variational approximations focusing on three concrete thrusts: (i) Statistical inference based on the Naive Mean Field (NMF) approximation for regression models, (ii) NMF approximation beyond regression and (iii) Advanced Mean Field approximations. Under theme (i), the PI will develop empirical Bayes methodology for the high-dimensional linear model, and compare Bayesian variable selection algorithms using the NMF approximation. Theme (ii) will focus on the NMF approximation for Hidden Markov Random Fields and Bayesian Neural Networks. Finally, theme (iii) will focus on certain alternative mean-field approximations. Physicists conjecture that if the number of datapoints and features are both large and comparable, the NMF approximation is no longer accurate; instead, the Thouless-Anderson-Palmer (TAP) approximation, an advanced mean-field approximation, should facilitate Bayes optimal inference. The proposed research will establish this conjecture in the context of high-dimensional linear regression under a proportional asymptotic regime. The theoretical foundations of the proposed methodology will rest on disparate ideas originating in non-linear large deviations (studied in probability and combinatorics), spin glasses (studied in probability and statistical physics) and graphical models. In turn, these ideas will be combined with classical statistical ideas (e.g. nonparametric maximum likelihood) to develop computationally efficient methods for high-dimensional inference. This cross-pollination of ideas will generate independent follow up research directions in each domain.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.