A nonparametric updating method to correct clinical prediction model drift
A nonparametric updating method to correct clinical prediction model drift
复制标题
DOI：
10.1093/jamia/ocz127
复制
发表时间：
2019-12-01
影响因子：
6.4
通讯作者：
Matheny, Michael E.
中科院分区：
文献类型：
--
作者：
Davis, Sharon E.;Greevy, Robert A., Jr.;Matheny, Michael E.
目的：临床预测模型需要更新，因为性能会随着时间的推移而恶化。我们开发了一个测试程序来选择更新方法，最大限度地减少过拟合，结合与更新样本大小的不确定性，并适用于参数和非参数models.Materials和方法：我们描述了一个程序来选择更新方法二分法的结果模型通过平衡简单性对准确性。我们说明了测试的属性的人口转移和2个模型的基础上模拟的情况下，退伍军人事务部住院admissions.Results：在模拟中，测试一般建议没有更新下没有人口转移，没有更新或适度的重新校准下的情况组合的变化，截距校正下不断变化的结果率，并根据改变预测结果协会改装。建议的更新提供了上级或类似的校准，实现了更复杂的更新。然而，在案例研究中，小更新集导致测试，建议简单的更新比可能已经理想的基础上随后performance.Discussion：我们的测试的建议强调了简单的更新，而不是系统的改装响应性能漂移的好处。正如预期的那样，建议的更新方法的复杂性反映了样本量和性能漂移的幅度。案例研究突出了我们的test.Conclusions的保守性：这个新的测试支持数据驱动的更新与生物统计和机器学习方法开发的模型，促进了广泛的临床预测模型的可移植性和维护，反过来，各种应用程序依赖于现代预测工具。
Objective: Clinical prediction models require updating as performance deteriorates over time. We developed a testing procedure to select updating methods that minimizes overfitting, incorporates uncertainty associated with updating sample sizes, and is applicable to both parametric and nonparametric models.Materials and Methods: We describe a procedure to select an updating method for dichotomous outcome models by balancing simplicity against accuracy. We illustrate the test's properties on simulated scenarios of population shift and 2 models based on Department of Veterans Affairs inpatient admissions.Results: In simulations, the test generally recommended no update under no population shift, no update or modest recalibration under case mix shifts, intercept correction under changing outcome rates, and refitting under shifted predictor-outcome associations. The recommended updates provided superior or similar calibration to that achieved with more complex updating. In the case study, however, small update sets lead the test to recommend simpler updates than may have been ideal based on subsequent performance.Discussion: Our test's recommendations highlighted the benefits of simple updating as opposed to systematic refitting in response to performance drift. The complexity of recommended updating methods reflected sample size and magnitude of performance drift, as anticipated. The case study highlights the conservative nature of our test.Conclusions: This new test supports data-driven updating of models developed with both biostatistical and machine learning approaches, promoting the transportability and maintenance of a wide array of clinical prediction models and, in turn, a variety of applications relying on modern prediction tools.