Massively Parallel Bayesian Inference Techniques

令人难以置信的广泛的研究领域依赖统计推断来获得和确认结果以及相应程度的确定性。常用的方法包括哈密顿蒙特卡罗(HMC)、马尔可夫链蒙特卡罗(MCMC)和重要性抽样。虽然MCMC和重要性抽样通常用于统计推断，通常执行速度很快，但在用于大型模型时，它们的伸缩性往往很差，无法收敛到正确的结果。另一方面，HMC往往更受欢迎，因为它与模型尺寸的伸缩性更好。然而，它往往很慢地收敛到结果(尽管是正确的)。我们的目标是提出贝叶斯推理方法，主要基于重要性抽样，使我们能够在合理的时间内在大型模型上得出正确的结果。由于这一项目的总体背景，其影响的潜在范围很广。特别是，统计推断通常由研究人员通过诸如STAN或PYMC等概率编程语言来执行。这个项目的一个长期目标是发布一种概率编程语言，它使用我们的“大规模并行”技术来快速而正确地执行这一推理，并具有熟悉的Pythonic界面。这些技术通过利用统计模型中的条件独立性，通过一种有效和易处理的方法有效地考虑指数数量的样本，从而广泛地发挥作用。这在实践中很快就完成了，这在很大程度上要归功于在GPU上执行所需操作时的并行性。正是出于这个原因，我们将我们的方法系列称为“大规模并行”方法。使用指数数量的样本(这里略有简化，以忽略某些交叉和冗余)与最近的结果一致，该结果表明，随着模型中变量的数量线性增加，重要性抽样工作所需的样本数量呈指数增加。这项工作提出了一种新的方法，可以应用于广泛的预先存在的推理方法，并已被证明在某些模型(特别是变量之间具有许多条件独立性的模型，如具有很大的层次结构)中取得了良好的结果。例如，重加权唤醒睡眠(RWS)算法的大规模并行版本(最初设计用于训练贝叶斯神经网络)已经开发出来，并被证明在几种情况下工作良好。除了概率编程语言的长期目标外，该项目更近期的目标是继续开发一种基于重要性加权后验矩估计的快速且正确的贝叶斯后验学习算法，并致力于将“大规模并行”方法引入其他推理算法。该项目属于EPSRC统计和应用概率研究领域。

An incredibly wide range of research areas rely on statistical inference to obtain and confirm results along with corresponding degrees of certainty. Popular techniques include Hamiltonian Monte Carlo (HMC), Markov Chain Monte Carlo (MCMC) and importance sampling. Whilst MCMC and importance sampling are commonly used for statistical inference, being generally fast to perform, they often scale poorly when used on large models, failing to converge to correct results. HMC, on the other hand, tends to be more popular as it scales better with model size. However, it is often slow to converge to the (albeit correct) results. We aim to present Bayesian inference methods, largely based on importance sampling, which allow us to arrive at correct results on large models in reasonable time. Due to the general setting of this project, there is a wide potential scope for its impact. In particular, statistical inference is often performed by researchers via probabilistic programming languages such as Stan or PyMC. One long term goal of this project is to release a probabilistic programming language which employs our "massively parallel" techniques to perform this inference both quickly and correctly, with a familiar Pythonic interface. These techniques work broadly by exploiting conditional independencies in statistical models through an efficient and tractable method for effectively considering an exponential number of samples. This is done quickly in practice thanks in large part to the parallelism available when performing the required operations on GPUs. It is for this reason that we refer to our family of methods as "massively parallel" methods. The use of an exponential number of samples (simplifying here somewhat to ignore certain crossovers and redundancies) aligns with a recent result which indicates that as the number of variables in a model increases linearly, the number of samples required for importance sampling to work increases exponentially. This work presents a novel approach that can be applied to a wide range of pre-existing inference methods and has already been shown to lead to favourable results in certain models (particularly models with many conditional independencies between variables, such as with a largely hierarchical structure). For instance, a massively parallel version of the Reweighted Wake-Sleep (RWS) algorithm (originally designed to train Bayesian neural networks) has already been developed and shown to work well in several cases. Alongside the long-term aim of a probabilistic programming language, the more immediate goals of this project are to continue developing an algorithm for fast and correct Bayesian posterior learning based on importance-weighted posterior moment estimates and to work on bringing the "massively parallel" approach to other inference algorithms. This project falls within the EPSRC Statistics and Applied Probability research area.