III: Small: Algorithms and Theoretical Foundations for Approximate Bayesian Inference in Machine Learning

在过去的二十年里，贝叶斯模型已经成为机器学习的核心。贝叶斯模型通常假设潜在的（未观察到的）变量对观察到的现象具有解释或预测能力。挑战在于从观察到的数据中推断这些变量的状态或对该状态的信念。例如，可以尝试通过观察用户自己的行为和其他用户的行为来推断用户的偏好。该项目的目标是开发跨大型贝叶斯模型家族的通用近似推理算法，以便解决方案可以广泛重用。算法工作将通过开发机器学习中近似贝叶斯推理的学习理论来补充。该理论框架旨在证明贝叶斯预测算法的性能保证，并为具有理想性能的算法设计提供信息。该项目将有助于基础科学研究，推进机器学习的核心目标。该项目将支持博士生的培训和研究，因此将直接支持人类发展。通过课堂教学和外展，该项目将使更多的学生了解机器学习及其应用潜力。更具体地说，该项目将研究非共轭贝叶斯潜变量模型，即，它将避免经常使用但限制简化的共轭假设。在算法方面，该项目将致力于推广非共轭图形模型的变分消息传递范式，并为此类模型的大子族开发使用最佳结构化近似的随机变分推理算法。提出的子族将捕捉到文献中许多重要问题的性质。在几个具体应用中的探索性研究进一步激励了这项工作，并将用于测试算法。该项目将为贝叶斯算法的理论分析开辟一个新的角度，在其预期误差的基础上推导出性能保证。一个核心思想是通过所谓的不可知论学习框架来看待变分推理算法，在这种框架中，所寻求的保证是相对于在特定有限的近似类别中可以做到的最佳结果。这将提供一个新的前景，通知算法的设计与期望的性能保证。该项目的预期科学影响是拥有更好的算法，具有良好的性能特征，并适用于更大类别的机器学习问题。

Over the last two decades Bayesian models have become central in machine learning. Bayesian models often hypothesize latent (non-observed) variables with explanatory or predictive power toward observed phenomena. The challenge is to infer the state of these variables or a belief over that state from observed data. For example, one might try to infer a user's preferences from observations about their own behavior and the behavior of other users. The goal of this project is to develop general approximate inference algorithms that work across large families of Bayesian models so that solutions can be widely reused. The algorithmic work will be complemented by developing a learning theory for approximate Bayesian inference in machine learning. The theoretical framework will aim to prove performance guarantees for Bayesian prediction algorithms and inform the design of algorithms with desirable properties. The project will contribute to basic scientific research, advancing core goals in machine learning. The project will support training and research of PhD students and therefore will directly support human development. Through classroom teaching and outreach the project will expose a larger population of students to machine learning and its potential in applications.More concretely, the project will investigate non-conjugate Bayesian latent variable models, i.e., it will avoid the often used but limiting simplifying assumption of conjugacy. On the algorithmic side the project will aim to generalize the paradigm of variational message passing for non-conjugate graphical models, and to develop stochastic variational inference algorithms using optimal structured approximations for large sub-families of such models. The proposed sub-families will capture the properties of many important problems in the literature. Exploratory research in several specific applications further motivates the work and will be used to test the algorithms. The project will develop a new angle for theoretical analysis of Bayesian algorithms, deriving performance guarantees on their expected error. A core idea is to view variational inference algorithms through the so-called agnostic learning framework where guarantees sought are relative to the best that can be done within a specific limited class of approximations. This will provide a fresh outlook that informs the design of algorithms with desired performance guarantees. The expected scientific impact of the project is having better algorithms with well understood performance characteristics and applicable for a larger class of machine learning problems.