Comparison of dynamic updating strategies for clinical prediction models.
Comparison of dynamic updating strategies for clinical prediction models.
复制标题
比较临床预测模型的动态更新策略。
DOI：
10.1186/s41512-021-00110-w
复制
发表时间：
2021-12-06
影响因子：
--
通讯作者：
Kimmel SE
中科院分区：
文献类型：
--
作者：
Schnellinger EM;Yang W;Kimmel SE
预测模型为许多医疗决策提供了信息，但它们的性能往往会随着时间的推移而恶化。文献中提出了几种离散时间更新策略，包括模型重新校准和修正。然而，这些策略尚未在动态更新环境下进行比较。我们使用了2010-2015年的肺移植后生存数据，并比较了以下更新策略的Brier评分（BS）、判别和校准：(1)从不更新，(2)使用文献中提出的封闭测试程序更新，(3)总是重新校准截距，(4)总是重新校准截距和斜率，(5)总是重新校准/修改模型。在每种情况下，我们探索了每1、2、4和8个季度的更新间隔。我们还研究了更新策略的性能如何随着更新中包含的旧数据量（即滑动窗口长度）的增加而变化。相对于从不更新，所有更新模型的方法都使BS得到了有意义的改善。无论采用何种更新策略，更频繁的更新都会产生更好的BS、判别和校准。与其他更新策略相比，重新校准策略带来了更一致的改进，并且随着时间的推移变化更少。使用更长的滑动窗口并没有实质性地影响重新校准策略，但确实改善了封闭测试程序和模型修正策略的区分和校准。模型更新导致改进的BS，更频繁的更新比不频繁的更新性能更好。模型重新校准策略对更新间隔和滑动窗口长度的敏感性最低。在线版本包含补充材料，可在10.1186/s41512-021-00110-w获得。
Prediction models inform many medical decisions, but their performance often deteriorates over time. Several discrete-time update strategies have been proposed in the literature, including model recalibration and revision. However, these strategies have not been compared in the dynamic updating setting. We used post-lung transplant survival data during 2010-2015 and compared the Brier Score (BS), discrimination, and calibration of the following update strategies: (1) never update, (2) update using the closed testing procedure proposed in the literature, (3) always recalibrate the intercept, (4) always recalibrate the intercept and slope, and (5) always refit/revise the model. In each case, we explored update intervals of every 1, 2, 4, and 8 quarters. We also examined how the performance of the update strategies changed as the amount of old data included in the update (i.e., sliding window length) increased. All methods of updating the model led to meaningful improvement in BS relative to never updating. More frequent updating yielded better BS, discrimination, and calibration, regardless of update strategy. Recalibration strategies led to more consistent improvements and less variability over time compared to the other updating strategies. Using longer sliding windows did not substantially impact the recalibration strategies, but did improve the discrimination and calibration of the closed testing procedure and model revision strategies. Model updating leads to improved BS, with more frequent updating performing better than less frequent updating. Model recalibration strategies appeared to be the least sensitive to the update interval and sliding window length. The online version contains supplementary material available at 10.1186/s41512-021-00110-w.