Dangers of Bayesian Model Averaging under Covariate Shift
Dangers of Bayesian Model Averaging under Covariate Shift
复制标题
DOI：
--
复制
发表时间：
2021-06
期刊：
影响因子：
--
通讯作者：
Pavel Izmailov;Patrick K. Nicholson;Sanae Lotfi;A. Wilson
中科院分区：
文献类型：
--
作者：
Pavel Izmailov;Patrick K. Nicholson;Sanae Lotfi;A. Wilson
神经网络的近似贝叶斯推理被认为是标准训练的一种稳健的替代方案，通常在非分布数据上提供良好的性能。然而，基于整批哈密顿蒙特卡罗的高保真近似推理的贝叶斯神经网络(BNN)在协变量漂移下的泛化性能较差，甚至低于经典估计。我们解释了这一令人惊讶的结果，展示了贝叶斯模型平均在协变量转移下实际上是如何存在问题的，特别是在输入特征中的线性依赖导致缺乏后验收缩的情况下。我们还说明了为什么相同的问题不会影响许多近似推理过程或经典的最大后验概率(MAP)训练。最后，我们提出了新的先验来提高BNN对多种协变量漂移来源的稳健性。
Approximate Bayesian inference for neural networks is considered a robust alternative to standard training, often providing good performance on out-of-distribution data. However, Bayesian neural networks (BNNs) with high-fidelity approximate inference via full-batch Hamiltonian Monte Carlo achieve poor generalization under covariate shift, even underperforming classical estimation. We explain this surprising result, showing how a Bayesian model average can in fact be problematic under covariate shift, particularly in cases where linear dependencies in the input features cause a lack of posterior contraction. We additionally show why the same issue does not affect many approximate inference procedures, or classical maximum a-posteriori (MAP) training. Finally, we propose novel priors that improve the robustness of BNNs to many sources of covariate shift.