Practical experiences on the necessity of external validation
Practical experiences on the necessity of external validation
复制标题
DOI：
10.1002/sim.3069
复制
发表时间：
2007-12-30
影响因子：
2
通讯作者：
Ziegler, A.
中科院分区：
文献类型：
--
作者：
Koenig, I. R.;Malley, J. D.;Ziegler, A.
预后模型的有效性是其在实际临床应用的重要前提。在这里，我们报告了一项针对中风患者的具体预后研究，并描述了我们如何探索我们的模型的预测性能。我们考虑了两个实际上高度相关的泛化方面，即模型在较晚时间点招募的患者中的表现（时间可运输性）和在不同于用于模型构建的医疗中心的表现（地理可运输性）。为了估计模型的准确性，我们研究了经典的内部验证技术和留一中心交叉验证（CV）。使用逻辑回归、支持向量机和随机森林（RFs）在训练集中建立预测脑卒中患者功能独立性的预后模型。采用十倍CV和留一中心CV来估计模型的时间和地理可迁移性。为了进行时间和外部验证，所得到的模型被用于对来自较晚时间点和不同诊所的患者进行分类。当应用回归模型或RFs时，可以很好地预测经典内部验证的时间验证数据的准确性。然而，当预测地理可运输性时，所有方法都存在困难。我们观察到，遗漏一个中心的CV比经典CV产生更好的估计。根据我们的结果，我们得出结论，在预后模型可以应用于实践之前，需要对来自不同诊所的患者进行外部验证。即使在晚些时候招募的患者中验证该模型，也不足以预测它在另一家诊所的表现。版权所有(C) 2007约翰威利父子有限公司
The validity of prognostic models is an important prerequisite for their applicability in practical clinical settings. Here, we report on a specific prognostic study on stroke patients and describe how we explored the prediction performance of our model. We considered two practically highly relevant generalization aspects, namely, the model's performance in patients recruited at a later time point (temporal transportability) and in medical centers different from those used for model building (geographic transportability). To estimate the accuracy of the model, we investigated classical internal validation techniques and leave-one-center-out cross validation (CV). Prognostic models predicting functional independence of stroke patients were developed in a training set using logistic regression, support vector machines, and random forests (RFs). Tenfold CV and leave-one-center-out CV were employed to estimate temporal and geographic transportability of the models. For temporal and external validation, the resulting models were used to classify patients from a later time point and from different clinics. When applying the regression model or the RFs, accuracy in the temporal validation data was well predicted from classical internal validation. However, when predicting geographic transportability all approaches had difficulties. We observed that the leave-one-center-out CV yielded better estimates than classical CV. On the basis of our results, we conclude that external validation in patients from different clinics is required before a prognostic model can be applied in practice. Even validating the model in patients recruited merely at a later time point does not suffice to predict how it may fare with regard to another clinic. Copyright (C) 2007 John Wiley & Sons, Ltd.