Model evaluation based on the sampling distribution of estimated absolute prediction error
Model evaluation based on the sampling distribution of estimated absolute prediction error
复制标题
DOI：
10.1093/biomet/asm036
复制
发表时间：
2007-06-01
期刊：
影响因子：
2.7
通讯作者：
Wei, L. J.
中科院分区：
文献类型：
--
作者：
Tian, Lu;Cai, Tianxi;Wei, L. J.
The construction of a reliable, practically useful prediction rule for future responses is heavily dependent on the 'adequacy' of the fitted regression model. In this article, we consider the absolute prediction error, the expected value of the absolute difference between the future and predicted responses, as the model evaluation criterion. This prediction error is easier to interpret than the average squared error and is equivalent to the misclassification error for a binary outcome. We show that the prediction error can be consistently estimated via the resubstitution and crossvalidation methods even when the fitted model is not correctly specified. Furthermore, we show that the resulting estimators are asymptotically normal. When the prediction rule is 'nonsmooth', the variance of the above normal distribution can be estimated well with a perturbation-resampling method. With two real examples and an extensive simulation study, we demonstrate that the interval estimates obtained from the above normal approximation for the prediction errors provide much more information about model adequacy than their point-estimate counterparts.
