Massively Parallel Bayesian Inference Techniques

点击翻译按钮获取中文摘要

An incredibly wide range of research areas rely on statistical inference to obtain and confirm results along with corresponding degrees of certainty. Popular techniques include Hamiltonian Monte Carlo (HMC), Markov Chain Monte Carlo (MCMC) and importance sampling. Whilst MCMC and importance sampling are commonly used for statistical inference, being generally fast to perform, they often scale poorly when used on large models, failing to converge to correct results. HMC, on the other hand, tends to be more popular as it scales better with model size. However, it is often slow to converge to the (albeit correct) results. We aim to present Bayesian inference methods, largely based on importance sampling, which allow us to arrive at correct results on large models in reasonable time. Due to the general setting of this project, there is a wide potential scope for its impact. In particular, statistical inference is often performed by researchers via probabilistic programming languages such as Stan or PyMC. One long term goal of this project is to release a probabilistic programming language which employs our "massively parallel" techniques to perform this inference both quickly and correctly, with a familiar Pythonic interface. These techniques work broadly by exploiting conditional independencies in statistical models through an efficient and tractable method for effectively considering an exponential number of samples. This is done quickly in practice thanks in large part to the parallelism available when performing the required operations on GPUs. It is for this reason that we refer to our family of methods as "massively parallel" methods. The use of an exponential number of samples (simplifying here somewhat to ignore certain crossovers and redundancies) aligns with a recent result which indicates that as the number of variables in a model increases linearly, the number of samples required for importance sampling to work increases exponentially. This work presents a novel approach that can be applied to a wide range of pre-existing inference methods and has already been shown to lead to favourable results in certain models (particularly models with many conditional independencies between variables, such as with a largely hierarchical structure). For instance, a massively parallel version of the Reweighted Wake-Sleep (RWS) algorithm (originally designed to train Bayesian neural networks) has already been developed and shown to work well in several cases. Alongside the long-term aim of a probabilistic programming language, the more immediate goals of this project are to continue developing an algorithm for fast and correct Bayesian posterior learning based on importance-weighted posterior moment estimates and to work on bringing the "massively parallel" approach to other inference algorithms. This project falls within the EPSRC Statistics and Applied Probability research area.