Improving Uncertainty Calibration of Deep Neural Networks via Truth Discovery and Geometric Optimization
Improving Uncertainty Calibration of Deep Neural Networks via Truth Discovery and Geometric Optimization
复制标题
DOI：
--
复制
发表时间：
2021-06
期刊：
影响因子：
--
通讯作者：
Chunwei Ma;Ziyun Huang;Jiayi Xian;Mingchen Gao;Jinhui Xu
中科院分区：
文献类型：
--
作者：
Chunwei Ma;Ziyun Huang;Jiayi Xian;Mingchen Gao;Jinhui Xu
深度神经网络（DNN）尽管近年来取得了巨大的成功，但由于其学习过程中固有的不确定性，仍然可能对其预测产生怀疑。Enhancement技术和事后校准是两种类型的方法，它们分别在改善DNN的不确定性校准方面表现出了希望。然而，这两类方法的协同效应尚未得到很好的探索。在本文中，我们提出了一个真理发现框架，集成基于集合和事后校准方法。使用集合候选的几何方差作为样本不确定性的一个很好的指标，我们设计了一个保持精度的真值估计，可证明没有精度下降。此外，我们还表明，事后校准也可以通过真理发现正则化优化来增强。在包括CIFAR和ImageNet在内的大规模数据集上，我们的方法在基于直方图和基于内核密度的评估指标上显示出与最先进的校准方法一致的改进。我们的代码可在https://github.com/horsepurve/truly-uncertain上获得。
Deep Neural Networks (DNNs), despite their tremendous success in recent years, could still cast doubts on their predictions due to the intrinsic uncertainty associated with their learning process. Ensemble techniques and post-hoc calibrations are two types of approaches that have individually shown promise in improving the uncertainty calibration of DNNs. However, the synergistic effect of the two types of methods has not been well explored. In this paper, we propose a truth discovery framework to integrate ensemble-based and post-hoc calibration methods. Using the geometric variance of the ensemble candidates as a good indicator for sample uncertainty, we design an accuracy-preserving truth estimator with provably no accuracy drop. Furthermore, we show that post-hoc calibration can also be enhanced by truth discovery-regularized optimization. On large-scale datasets including CIFAR and ImageNet, our method shows consistent improvement against state-of-the-art calibration approaches on both histogram-based and kernel density-based evaluation metrics. Our codes are available at https://github.com/horsepurve/truly-uncertain.