External validation of multivariable prediction models: a systematic review of methodological conduct and reporting.
External validation of multivariable prediction models: a systematic review of methodological conduct and reporting.
复制标题
DOI：
10.1186/1471-2288-14-40
复制
发表时间：
2014-03-19
影响因子：
4
通讯作者：
Altman DG
中科院分区：
文献类型：
--
作者：
Collins GS;de Groot JA;Dutton S;Omar O;Shanyinde M;Tajar A;Voysey M;Wharton R;Yu LM;Moons KG;Altman DG
在考虑是否使用多变量（诊断或预后）预测模型之前，必须在未用于开发模型的数据中评估其性能（称为外部验证）。我们批判性地评价了多变量预测模型外部验证研究的方法学实施和报告。我们对2010年发表的PubMed核心临床期刊中描述一个或多个多变量预测模型的某种形式的外部验证的文章进行了系统性综述。在设计、样本量、缺失数据处理、参考原始研究开发预测模型和预测性能指标方面，一式两份提取研究数据。共识别出11，826篇文章，其中78篇被纳入全面综述，描述了120种预测模型的评价。参与者的数据没有被用来开发模型。33篇文章描述了预测模型的开发及其在单独数据集上的性能评估，45篇文章仅描述了对另一个数据集的现有已发表预测模型的评估。57%的预测模型作为简化的评分系统进行了介绍和评估。16%的文章未能报告验证数据集中的结局事件数量。54%的研究没有明确提到缺失数据。67%的研究没有报告评估模型校准，而大多数研究评估了模型区分度。通常不清楚所报告的业绩计量是针对完整回归模型还是针对简化模型。绝大多数描述多变量预测模型的某种形式的外部验证的研究报告很少，关键细节往往没有提出。验证研究的特点是设计不良、处理不当和确认缺失数据，以及预测模型最关键的性能指标之一，即经常从出版物中遗漏的校准。因此，当缺乏进行良好和明确报告（外部验证）的研究来描述其对独立参与者数据的性能时，绝大多数已开发的预测模型在实践中没有使用就不足为奇了。
Before considering whether to use a multivariable (diagnostic or prognostic) prediction model, it is essential that its performance be evaluated in data that were not used to develop the model (referred to as external validation). We critically appraised the methodological conduct and reporting of external validation studies of multivariable prediction models. We conducted a systematic review of articles describing some form of external validation of one or more multivariable prediction models indexed in PubMed core clinical journals published in 2010. Study data were extracted in duplicate on design, sample size, handling of missing data, reference to the original study developing the prediction models and predictive performance measures. 11,826 articles were identified and 78 were included for full review, which described the evaluation of 120 prediction models. in participant data that were not used to develop the model. Thirty-three articles described both the development of a prediction model and an evaluation of its performance on a separate dataset, and 45 articles described only the evaluation of an existing published prediction model on another dataset. Fifty-seven percent of the prediction models were presented and evaluated as simplified scoring systems. Sixteen percent of articles failed to report the number of outcome events in the validation datasets. Fifty-four percent of studies made no explicit mention of missing data. Sixty-seven percent did not report evaluating model calibration whilst most studies evaluated model discrimination. It was often unclear whether the reported performance measures were for the full regression model or for the simplified models. The vast majority of studies describing some form of external validation of a multivariable prediction model were poorly reported with key details frequently not presented. The validation studies were characterised by poor design, inappropriate handling and acknowledgement of missing data and one of the most key performance measures of prediction models i.e. calibration often omitted from the publication. It may therefore not be surprising that an overwhelming majority of developed prediction models are not used in practice, when there is a dearth of well-conducted and clearly reported (external validation) studies describing their performance on independent participant data.
登录
查看更多内容
DOI：
10.1126/science.1211811
发表时间：
2011-12-02
期刊：
Science (New York, N.Y.)
影响因子：
--
作者：
Ioannidis JP;Khoury MJ
通讯作者：
Khoury MJ
DOI：
10.1136/bmj.d3651
发表时间：
2011-06-22
期刊：
BMJ (Clinical research ed.)
影响因子：
--
作者：
Collins GS;Mallett S;Altman DG
通讯作者：
Altman DG
影响因子：
4.7
作者：
Ioannidis, John P. A.
通讯作者：
Ioannidis, John P. A.
影响因子：
3.5
作者：
Jaja, Blessing N. R.;Cusimano, Michael D.;Macdonald, R. Loch
通讯作者：
Macdonald, R. Loch
DOI：
10.1111/j.1467-985x.2009.00592.x
发表时间：
2009-01-01
影响因子：
2
作者：
Baker, Stuart G.;Cook, Nancy R.;Kramer, Barnett S.
通讯作者：
Kramer, Barnett S.