III: Small: Algorithms and Theoretical Foundations for Approximate Bayesian Inference in Machine Learning

在过去的二十年里，贝叶斯模型已经成为机器学习的核心。贝叶斯模型经常假设对观察到的现象具有解释或预测能力的潜在(未观察到)变量。挑战是从观察到的数据中推断出这些变量的状态或对该状态的看法。例如，可以尝试通过观察用户自己的行为和其他用户的行为来推断用户的偏好。该项目的目标是开发适用于贝叶斯模型大家族的通用近似推理算法，以便解决方案可以广泛重复使用。算法工作将通过开发机器学习中的近似贝叶斯推理的学习理论来补充。该理论框架将旨在证明贝叶斯预测算法的性能保证，并为具有理想特性的算法的设计提供指导。该项目将为基础科学研究做出贡献，推进机器学习的核心目标。该项目将支持博士生的培训和研究，因此将直接支持人的发展。通过课堂教学和推广，该项目将使更多的学生接触到机器学习及其在应用中的潜力。更具体地说，该项目将研究非共轭贝叶斯潜变量模型，即它将避免经常使用但有限的简化共轭假设。在算法方面，该项目将致力于推广非共轭图形模型的变分消息传递范例，并开发用于此类模型的大子族的使用最优结构近似的随机变分推理算法。提出的子族将捕捉到文献中许多重要问题的属性。在几个具体应用中的探索性研究进一步推动了这项工作，并将用于测试算法。该项目将为贝叶斯算法的理论分析开辟一个新的角度，得到其预期误差的性能保证。一个核心思想是通过所谓的不可知性学习框架来看待变分推理算法，在该框架中，所寻求的保证相对于在特定有限的近似类别中所能完成的最好。这将提供一个全新的前景，为算法的设计提供所需的性能保证。该项目的预期科学影响是拥有更好的算法，具有很好的性能特征，并适用于更大类别的机器学习问题。

Over the last two decades Bayesian models have become central in machine learning. Bayesian models often hypothesize latent (non-observed) variables with explanatory or predictive power toward observed phenomena. The challenge is to infer the state of these variables or a belief over that state from observed data. For example, one might try to infer a user's preferences from observations about their own behavior and the behavior of other users. The goal of this project is to develop general approximate inference algorithms that work across large families of Bayesian models so that solutions can be widely reused. The algorithmic work will be complemented by developing a learning theory for approximate Bayesian inference in machine learning. The theoretical framework will aim to prove performance guarantees for Bayesian prediction algorithms and inform the design of algorithms with desirable properties. The project will contribute to basic scientific research, advancing core goals in machine learning. The project will support training and research of PhD students and therefore will directly support human development. Through classroom teaching and outreach the project will expose a larger population of students to machine learning and its potential in applications.More concretely, the project will investigate non-conjugate Bayesian latent variable models, i.e., it will avoid the often used but limiting simplifying assumption of conjugacy. On the algorithmic side the project will aim to generalize the paradigm of variational message passing for non-conjugate graphical models, and to develop stochastic variational inference algorithms using optimal structured approximations for large sub-families of such models. The proposed sub-families will capture the properties of many important problems in the literature. Exploratory research in several specific applications further motivates the work and will be used to test the algorithms. The project will develop a new angle for theoretical analysis of Bayesian algorithms, deriving performance guarantees on their expected error. A core idea is to view variational inference algorithms through the so-called agnostic learning framework where guarantees sought are relative to the best that can be done within a specific limited class of approximations. This will provide a fresh outlook that informs the design of algorithms with desired performance guarantees. The expected scientific impact of the project is having better algorithms with well understood performance characteristics and applicable for a larger class of machine learning problems.