A Rigorous Link between Deep Ensembles and (Variational) Bayesian Methods
A Rigorous Link between Deep Ensembles and (Variational) Bayesian Methods
复制标题
DOI：
10.48550/arxiv.2305.15027
复制
发表时间：
2023-05
期刊：
影响因子：
--
通讯作者：
Veit Wild;Sahra Ghalebikesabi;D. Sejdinovic;Jeremias Knoblauch
中科院分区：
文献类型：
--
作者：
Veit Wild;Sahra Ghalebikesabi;D. Sejdinovic;Jeremias Knoblauch
我们建立了贝叶斯，变分贝叶斯和合奏方法之间的第一个数学上严格的联系。实现这一目标的关键一步是将深度学习中通常遇到的非凸优化问题重新表述为概率测度空间中的凸优化。在技术层面上，我们的贡献相当于研究广义变分推理，通过Wasserstein梯度流。其结果是各种看似无关的方法的统一理论，这些方法通常用于深度学习中的不确定性量化-包括深度集成和（变分）贝叶斯方法。这提供了一个新的角度对深集成的成功背后的原因参数化变分推理的基础上的程序，并允许导出新的集成方案的收敛保证。我们展示了这一点，提出了一个家庭的相互作用的深合奏与直接平行的粒子系统在热力学中的相互作用，并使用我们的理论来证明这些算法的收敛到一个定义良好的全局极小的概率测度空间。
We establish the first mathematically rigorous link between Bayesian, variational Bayesian, and ensemble methods. A key step towards this it to reformulate the non-convex optimisation problem typically encountered in deep learning as a convex optimisation in the space of probability measures. On a technical level, our contribution amounts to studying generalised variational inference through the lense of Wasserstein gradient flows. The result is a unified theory of various seemingly disconnected approaches that are commonly used for uncertainty quantification in deep learning -- including deep ensembles and (variational) Bayesian methods. This offers a fresh perspective on the reasons behind the success of deep ensembles over procedures based on parameterised variational inference, and allows the derivation of new ensembling schemes with convergence guarantees. We showcase this by proposing a family of interacting deep ensembles with direct parallels to the interactions of particle systems in thermodynamics, and use our theory to prove the convergence of these algorithms to a well-defined global minimiser on the space of probability measures.