Individual Calibration with Randomized Forecasting
Individual Calibration with Randomized Forecasting
复制标题
DOI：
--
复制
发表时间：
2020-06
期刊：
影响因子：
--
通讯作者：
Shengjia Zhao;Tengyu Ma;Stefano Ermon
中科院分区：
文献类型：
--
作者：
Shengjia Zhao;Tengyu Ma;Stefano Ermon
机器学习应用程序通常需要校准预测，例如90%的可信区间应该包含90%的真实结果。然而，典型的校准定义仅要求其保持平均值，并不保证对单个样本进行预测。因此，预测可能会在某些子组上系统地过度或不足，导致公平性和潜在漏洞的问题。我们表明，如果预测是随机的，即输出随机可信区间，则在回归设置中可以对单个样本进行校准。随机化通过权衡偏差和方差来消除系统性偏差。我们设计了一个训练目标来执行个体校准，并使用它来训练随机回归函数。由此产生的模型对任意选择的数据子组进行了更多的校准，并且可以在针对利用错误校准预测的对手的决策中实现更高的实用性。
Machine learning applications often require calibrated predictions, e.g. a 90\% credible interval should contain the true outcome 90\% of the times. However, typical definitions of calibration only require this to hold on average, and offer no guarantees on predictions made on individual samples. Thus, predictions can be systematically over or under confident on certain subgroups, leading to issues of fairness and potential vulnerabilities. We show that calibration for individual samples is possible in the regression setup if the predictions are randomized, i.e. outputting randomized credible intervals. Randomization removes systematic bias by trading off bias with variance. We design a training objective to enforce individual calibration and use it to train randomized regression functions. The resulting models are more calibrated for arbitrarily chosen subgroups of the data, and can achieve higher utility in decision making against adversaries that exploit miscalibrated predictions.