The Integrated Calibration Index (ICI) and related metrics for quantifying the calibration of logistic regression models
The Integrated Calibration Index (ICI) and related metrics for quantifying the calibration of logistic regression models
复制标题
DOI：
10.1002/sim.8281
复制
发表时间：
2019-09-20
影响因子：
2
通讯作者：
Steyerberg, Ewout W.
中科院分区：
文献类型：
--
作者：
Austin, Peter C.;Steyerberg, Ewout W.
评估用于估计二元结果发生概率的方法的校准是验证风险预测算法性能的一个重要方面。校准通常指的是预测和观察到的结果概率之间的一致性。图形方法是一种有吸引力的方法来评估校准，其中观察到的和预测的概率进行比较，使用黄土为基础的平滑函数。我们描述了综合校准指数（ICI），这是由Harrell的E-max指数，这是一个平滑的校准曲线和完美的校准的对角线之间的最大绝对差异的动机。ICI可以解释为观测概率和预测概率之间的加权差，其中观测值由预测概率的经验密度函数加权。因此，ICI是一种明确包含预测概率分布的校准度量。我们还讨论了两个相关的校准措施，E50和E90，这代表了观察和预测概率之间的绝对差异的中位数和第90百分位数。我们通过使用ICI，E50和E90来比较Logistic回归与随机森林和增强回归树的校准来说明ICI，E50和E90的实用性，以预测心脏病发作住院患者的死亡率。使用这些数字度量允许在校准中比通过目视检查图形校准曲线所允许的更大的差异。
Assessing the calibration of methods for estimating the probability of the occurrence of a binary outcome is an important aspect of validating the performance of risk-prediction algorithms. Calibration commonly refers to the agreement between predicted and observed probabilities of the outcome. Graphical methods are an attractive approach to assess calibration, in which observed and predicted probabilities are compared using loess-based smoothing functions. We describe the Integrated Calibration Index (ICI) that is motivated by Harrell's E-max index, which is the maximum absolute difference between a smooth calibration curve and the diagonal line of perfect calibration. The ICI can be interpreted as weighted difference between observed and predicted probabilities, in which observations are weighted by the empirical density function of the predicted probabilities. As such, the ICI is a measure of calibration that explicitly incorporates the distribution of predicted probabilities. We also discuss two related measures of calibration, E50 and E90, which represent the median and 90th percentile of the absolute difference between observed and predicted probabilities. We illustrate the utility of the ICI, E50, and E90 by using them to compare the calibration of logistic regression with that of random forests and boosted regression trees for predicting mortality in patients hospitalized with a heart attack. The use of these numeric metrics permitted for a greater differentiation in calibration than was permissible by visual inspection of graphical calibration curves.