Fast Bayesian Inference at Extreme Scale

本研究旨在推动人工智能方法的发展，为预测提供准确的预测和高质量的不确定性表示。贝叶斯推理算法的统计家族很好地满足了这两个要求，但要将这些算法扩展到“极端规模”，同时保持高精度和校准好的不确定性测量，还需要做更多的工作。两种著名的贝叶斯推理算法是高斯过程（gp）和贝叶斯神经网络（bnn）。GPs提供了原则良好的不确定性表示，但其精确形式不能很好地扩展——尽管最近在这个方向上有了很好的进展，使用了百万数据点级别的新型分布式线性代数实现。这项研究将面向具有100万数据点以上的应用程序，这是精确GP目前所面临的困难。为了处理这些缩放困难，已经开发了各种gp近似，所有这些近似都涉及预测质量和实现速度之间的权衡。撇开缩放困难不谈，gp提供的有效不确定性表示是非常有利的。但当我们研究深度神经网络时，这种情况就完全不同了。如果我们抛开对不确定性测量的任何要求，那么深度神经网络可以很好地扩展到极端规模。然而，可靠的不确定性表示很难通过深度学习和贝叶斯神经网络（BNN）来实现。在此背景下，研究目标是扩展大规模贝叶斯推理算法的最新技术，旨在(1)提高100万以上数据集的速度（并具有良好的可扩展性特征）；(2)在预测精度和提供校准良好的不确定性表示方面提高可靠性(3)随着数据集大小的增加，用户友好的调整以在模型性能和训练/测试成本之间取得最佳平衡。

This research aims to advance the state-of-the-art in those AI methodologies which furnish both accurate predictions and high-quality uncertainty representations for the predictions made. The statistical family of Bayesian inference algorithms is well placed to deliver both of these requirements, but more needs to be done to extend these algorithms to 'extreme scale' whilst at the same time maintaining high accuracy and well calibrated measures of uncertainty. Two notable Bayesian inference algorithms are Gaussian processes (GPs) and Bayesian Neural Networks (BNNs). GPs provide well-principled uncertainty representations but in their exact form do not scale well - although there has been recent good progress in this direction using novel distributed linear algebra implementations at the million-data-point level. This research will be geared toward applications with the order of a million data-points plus, which is where an exact GP currently struggles. To deal with these scaling difficulties various approximations to GPs have been developed, all involving a trade-off between quality of prediction and speed of implementation. Setting aside scaling difficulties, the effective uncertainty representations provided by GPs are extremely advantageous. But this situation is turned on its head when we look at deep neural networks. If we set aside any requirement for uncertainty measures then deep neural networks scale very well to extreme scale. However, reliable uncertainty representations are much harder to come by with deep learning and Bayesian Neural Networks (BNN). Given this background the research aim is to extend the state-of-the-art in large scale Bayesian inference algorithms, aiming for (1) improvements in speeds on data-sets of size one million plus (and with good scalability characteristics beyond) ; (2) improvements in reliability both in terms of prediction accuracy and provision of well calibrated uncertainty representations (3) user friendly tuning to strike an optimised balance between model performance and training/testing costs as the data-set sizes increases.