Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration
Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration
复制标题
DOI：
--
复制
发表时间：
2019-09
期刊：
影响因子：
--
通讯作者：
Meelis Kull;Miquel Perello-Nieto;Markus Kängsepp;Telmo de Menezes e Silva Filho;Hao Song;Peter A. Flach-
中科院分区：
文献类型：
--
作者：
Meelis Kull;Miquel Perello-Nieto;Markus Kängsepp;Telmo de Menezes e Silva Filho;Hao Song;Peter A. Flach-
大多数多类分类器预测的类概率是未经校准的，往往倾向于过度自信。使用神经网络，可以通过温度缩放来改进校准，这是一种学习最后softmax层输入的单个校正乘法因子的方法。在非神经模型上，现有方法以成对或一对一的方式应用二进制校准。我们提出了一种适用于任何模型类的分类器的原生多类校准方法，该方法来自Dirichlet分布，并从二进制分类中推广了Beta校准方法。它很容易用神经网络实现，因为它相当于对未校准的概率进行对数变换，然后是一个线性层和softmax。实验表明，改进的概率预测，根据多种措施（置信ECE，classwise-ECE，日志损失，Brier得分）在广泛的数据集和分类。学习的狄利克雷校准图的参数提供了对未校准模型中的偏差的见解。
Class probabilities predicted by most multiclass classifiers are uncalibrated, often tending towards over-confidence. With neural networks, calibration can be improved by temperature scaling, a method to learn a single corrective multiplicative factor for inputs to the last softmax layer. On non-neural models the existing methods apply binary calibration in a pairwise or one-vs-rest fashion. We propose a natively multiclass calibration method applicable to classifiers from any model class, derived from Dirichlet distributions and generalising the beta calibration method from binary classification. It is easily implemented with neural nets since it is equivalent to log-transforming the uncalibrated probabilities, followed by one linear layer and softmax. Experiments demonstrate improved probabilistic predictions according to multiple measures (confidence-ECE, classwise-ECE, log-loss, Brier score) across a wide range of datasets and classifiers. Parameters of the learned Dirichlet calibration map provide insights to the biases in the uncalibrated model.