Accommodating population differences when validating risk prediction models.
Accommodating population differences when validating risk prediction models.
复制标题
DOI：
10.1002/sim.9447
复制
发表时间：
2022-10-30
影响因子：
2
通讯作者：
Ankerst, Donna P.
中科院分区：
文献类型：
--
作者：
Pfeiffer, Ruth M.;Chen, Yiyao;Gail, Mitchell H.;Ankerst, Donna P.
在独立数据中验证风险预测模型提供了比内部评估更严格的模型性能评估，例如通过对用于模型开发的数据进行交叉验证。然而，导致培训和验证数据的总体之间的几个差异可能会导致风险模型的表现看起来很差。在本文中，我们形式化了训练和验证数据的“相似性”或“相关性”的概念，并定义了可重复性和可移植性。我们讨论了模型预测因子的不同分布以及在验证疾病状态或结果时的差异对模型的校准、准确性和判别度的测量的影响。当来自训练和验证数据集的个人水平信息可用时，我们提出并研究验证度量的加权版本，该加权版本调整训练和验证数据之间的风险因素分布和结果验证的差异，以提供对模型性能的更全面的评估。我们提供了风险模型和产生训练和验证数据的人群的条件，以确保模型的重复性或可移植性，并展示了如何使用加权和未加权的性能度量来检查这些条件。我们通过开发和验证一个模型来说明该方法，该模型使用两个大型前列腺癌筛查试验的数据来预测发生前列腺癌的风险。
Validation of risk prediction models in independent data provides a more rigorous assessment of model performance than internal assessment, e.g. done by crossvalidation in the data used for model development. However, several differences between the populations that gave rise to the training and the validation data can lead to seemingly poor performance of a risk model. In this paper we formalize the notions of “similarity” or “relatedness” of the training and validation data, and define reproducibility and transportability. We address the impact of different distributions of model predictors and differences in verifying the disease status or outcome on measures of calibration, accuracy and discrimination of a model. When individual level information from both the training and validation data sets is available, we propose and study weighted versions of the validation metrics that adjust for differences in the risk factor distributions and in outcome verification between the training and validation data to provide a more comprehensive assessment of model performance. We provide conditions on the risk model and the populations that gave rise to the training and validation data that ensure a model’s reproducibility or transportability, and show how to check these conditions using weighted and unweighted performance measures. We illustrate the method by developing and validating a model that predicts the risk of developing prostate cancer using data from two large prostate cancer screening trials.
登录
查看更多内容
影响因子：
16.6
作者：
Song X;Yu ASL;Kellum JA;Waitman LR;Matheny ME;Simpson SQ;Hu Y;Liu M
通讯作者：
Liu M
影响因子：
38.9
作者：
Metnitz, PGH;Lang, T;Le Gall, JR
通讯作者：
Le Gall, JR
影响因子：
2.7
作者：
Cook, ED;Moody-Thomas, S;Probstfield, JL
通讯作者：
Probstfield, JL
影响因子：
2
作者：
Steyerberg, Ewout W.;Nieboer, Daan;van Houwelingen, Hans C.
通讯作者：
van Houwelingen, Hans C.
DOI：
10.1111/j.1467-9876.2005.00477.x
发表时间：
2005-01-01
影响因子：
1.6
作者：
Alonzo, TA;Pepe, MS
通讯作者：
Pepe, MS