Random Matrix Approaches to Approximate Bayesian Inference in Machine Learning

贝叶斯范式为从数据中学习提供了重要的方法。它将用于生成数据的概率模型与关于模型参数的概率分布内的可能参数的先验知识结合在一起。然而，这一思想在具有大量参数的模型中的实际应用经常受到与高维概率分布的难解性相关的计算问题的困扰。机器学习的近似推理方法提供了用更简单的分布来近似这种分布的算法-通常是通过多变量高斯分布。这些推理方法在应用中往往会产生很好的结果。但是，在迭代推理算法中更新这些高斯分布的协方差矩阵(它提供了关于变量之间的不确定性和相关性的重要信息)需要每次迭代的矩阵运算，这使得模型的参数数量增加到立方。这使得这种方法的应用在变量数量较大时存在问题。因此，进一步的近似是必要的。而这些可能会降低预测的质量。第二个相关问题是，一些流行的推理算法不能保证收敛。目前尚不清楚未能收敛是算法的伪影，还是与贝叶斯模型的复杂性有关。在信息论和统计物理领域最新研究的推动下，该项目将从一个新的角度解决这些问题。假设数据矩阵可以被认为是随机的(以数学上定义良好的方式)，随机矩阵理论的结果提出了有效地近似所需矩阵运算的新方法。当矩阵较大时，这些逼近有望在渐近极限内表现良好。随机矩阵方法还将为在一定的数据统计假设下分析大型问题的迭代推理算法的性能提供新的途径。我们将使用这些随机矩阵技术来加快现有算法的速度，并设计具有优化收敛特性的新算法。我们将调查这些方法的质量和稳健性。最后，我们将在机器学习中的各种贝叶斯模型上验证我们的方法，并在模拟和真实数据上与竞争方法的性能进行比较。

The Bayesian paradigm provides important methods for learning from data. It combines a probabilistic model for the generation of data together with prior knowledge over likely parameters within a probability distribution over the parameters of the model. However, practical applications of this idea to models with a large number of parameters are often plagued by computational problems related to the intractability of high--dimensional probability distributions. Approximate inference methods of machine learning provide algorithms for approximating such distributions by simpler ones - typically by multivariate Gaussian distributions. These inference methods yield often excellent results in applications. But, the update of covariance matrices (which give important information on uncertainties and dependencies between variables) of these Gaussian distributions within the iterative inference algorithms requires matrix operations per iteration which grows cubic in the number of parameters of the model. This makes the applications of such methods problematic when the number of variables is large. Hence, further approximations are necessary. And these may deteriorate the quality of the predictions. A second relevant problem is the fact that there is no guarantee of convergence for some popular inference algorithms. It is unclear if the failure to converge is an artefact of the algorithm or is related to the complexity of the Bayesian model. Motivated by recent research in the fields of information theory and statistical physics, this project will address these problems from a new angle. Assuming that data matrices can be considered as random (in a mathematically well-defined way), results of random matrix theory suggest novel ways to efficiently approximate the required matrix operations. These approximations are expected to perform well in the asymptotic limit when matrices are large. Random matrix methods will also provide new ways for analyzing the performance of iterative inference algorithms for large problems under certain statistical assumptions on the data. We will use these random matrix techniques to speed up existing algorithms as well as designing novel algorithms with optimized convergence properties. We will investigate the quality and robustness of such methods. Finally, we will validate our approach on various Bayesian models in machine learning and compare the performance with that of competing methods on simulated as well as real data.