Deterministic Variational Inference for Robust Bayesian Neural Networks
Deterministic Variational Inference for Robust Bayesian Neural Networks
复制标题
DOI：
--
复制
发表时间：
2018-09
期刊：
影响因子：
--
通讯作者：
Anqi Wu;Sebastian Nowozin;Edward Meeds;Richard E. Turner;José Miguel Hernández-Lobato;Alexander L. Gaunt
中科院分区：
文献类型：
--
作者：
Anqi Wu;Sebastian Nowozin;Edward Meeds;Richard E. Turner;José Miguel Hernández-Lobato;Alexander L. Gaunt
贝叶斯神经网络(BNN)作为一种灵活的、原则性的解决方案，在处理从有限数据中学习时的不确定性方面有着很大的潜力。在深度神经网络中实现概率推理的方法中，变分贝叶斯(VB)具有理论基础、普遍适用性和计算效率等优点。随着人们对潜在优势的广泛认识，为什么变分贝叶斯在实际应用中看到的BNN的实际应用非常有限？我们认为神经网络中的变分推理是脆弱的：成功的实现需要仔细地初始化和调整先验方差，以及控制蒙特卡罗梯度估计的方差。我们提供了两个创新，旨在使VB成为贝叶斯神经网络的稳健推理工具：第一，我们引入了一种新的确定性方法来逼近神经网络中的矩，消除了梯度方差；第二，我们引入了参数的分层先验和一种新的经验贝叶斯方法来自动选择先验方差。将这两个创新结合在一起，得到的方法是高效和健壮的。在异方差回归的应用上，我们表现出了比其他方法更好的预测性能。
Bayesian neural networks (BNNs) hold great promise as a flexible and principled solution to deal with uncertainty when learning from finite data. Among approaches to realize probabilistic inference in deep neural networks, variational Bayes (VB) is theoretically grounded, generally applicable, and computationally efficient. With wide recognition of potential advantages, why is it that variational Bayes has seen very limited practical use for BNNs in real applications? We argue that variational inference in neural networks is fragile: successful implementations require careful initialization and tuning of prior variances, as well as controlling the variance of Monte Carlo gradient estimates. We provide two innovations that aim to turn VB into a robust inference tool for Bayesian neural networks: first, we introduce a novel deterministic method to approximate moments in neural networks, eliminating gradient variance; second, we introduce a hierarchical prior for parameters and a novel Empirical Bayes procedure for automatically selecting prior variances. Combining these two innovations, the resulting method is highly efficient and robust. On the application of heteroscedastic regression we demonstrate good predictive performance over alternative approaches.