Variational Inference on the Final-Layer Output of Neural Networks
Variational Inference on the Final-Layer Output of Neural Networks
复制标题
DOI：
--
复制
发表时间：
2023-02
期刊：
影响因子：
--
通讯作者：
Yadi Wei;R. Khardon
中科院分区：
文献类型：
--
作者：
Yadi Wei;R. Khardon
传统的神经网络很容易训练，但它们通常会产生过于自信的预测。相比之下，贝叶斯神经网络提供了良好的不确定性量化，但由于参数空间较大，优化它们非常耗时。本文提出了联合收割机的优点，这两种方法进行变分推理的最终层输出空间（VIFO），因为输出空间是远远小于参数空间。我们使用神经网络来学习概率输出的均值和方差。与标准的非Bestmann模型一样，VIFO具有简单的训练，并且可以使用Rademacher复杂度为模型提供风险边界。另一方面，使用贝叶斯公式，我们将塌陷变分推理与VIFO结合起来，显着提高了实践中的性能。实验表明，VIFO和VIFO的集成提供了一个很好的折衷的运行时间和不确定性量化，特别是对于分布数据。
Traditional neural networks are simple to train but they typically produce overconfident predictions. In contrast, Bayesian neural networks provide good uncertainty quantification but optimizing them is time consuming due to the large parameter space. This paper proposes to combine the advantages of both approaches by performing Variational Inference in the Final layer Output space (VIFO), because the output space is much smaller than the parameter space. We use neural networks to learn the mean and the variance of the probabilistic output. Like standard, non-Beyesian models, VIFO enjoys simple training and one can use Rademacher complexity to provide risk bounds for the model. On the other hand, using the Bayesian formulation we incorporate collapsed variational inference with VIFO which significantly improves the performance in practice. Experiments show that VIFO and ensembles of VIFO provide a good tradeoff in terms of run time and uncertainty quantification, especially for out of distribution data.