Calibrated and Sharp Uncertainties in Deep Learning via Density Estimation
Calibrated and Sharp Uncertainties in Deep Learning via Density Estimation
复制标题
DOI：
--
复制
发表时间：
2021-12
期刊：
影响因子：
--
通讯作者：
Volodymyr Kuleshov;Shachi Deshpande
中科院分区：
文献类型：
--
作者：
Volodymyr Kuleshov;Shachi Deshpande
准确的概率预测可以通过两个属性来表征——校准和清晰度。然而，标准最大似然训练生成的模型校准不佳，因此不准确——90% 的置信区间通常在 90% 的情况下并不包含真实结果。本文认为校准在实践中很重要，并且通过执行低维密度估计很容易维护。我们引入了一个基于重新校准的简单训练程序，可以在不牺牲整体性能的情况下生成校准模型；与以前的方法不同，我们的方法确保了分布校准的最通用属性，并适用于任何模型，包括神经网络。我们正式证明了我们的程序的正确性，假设我们可以估计低维度的密度并建立统一的收敛边界。我们的结果对线性和深度贝叶斯模型产生了实证性能改进，并表明应该在机器学习中越来越多地利用校准。
Accurate probabilistic predictions can be characterized by two properties -- calibration and sharpness. However, standard maximum likelihood training yields models that are poorly calibrated and thus inaccurate -- a 90% confidence interval typically does not contain the true outcome 90% of the time. This paper argues that calibration is important in practice and is easy to maintain by performing low-dimensional density estimation. We introduce a simple training procedure based on recalibration that yields calibrated models without sacrificing overall performance; unlike previous approaches, ours ensures the most general property of distribution calibration and applies to any model, including neural networks. We formally prove the correctness of our procedure assuming that we can estimate densities in low dimensions and we establish uniform convergence bounds. Our results yield empirical performance improvements on linear and deep Bayesian models and suggest that calibration should be increasingly leveraged across machine learning.