Penalization and shrinkage methods produced unreliable clinical prediction models especially when sample size was small.
Penalization and shrinkage methods produced unreliable clinical prediction models especially when sample size was small.
复制标题
DOI：
10.1016/j.jclinepi.2020.12.005
复制
发表时间：
2021-04
影响因子：
7.2
通讯作者：
Collins GS
中科院分区：
文献类型：
--
作者：
Riley RD;Snell KIE;Martin GP;Whittle R;Archer L;Sperrin M;Collins GS
在开发临床预测模型时，建议使用惩罚技术来解决过拟合问题，因为它们将预测效应估计值缩小到零值，并减少新个体的均方预测误差。然而，收缩和惩罚条款（'调整参数'）估计与不确定性的发展数据集。我们研究了这种不确定性的大小以及对预测模型性能的后续影响。本研究包括以下方法的应用实例和模拟研究：均匀收缩（通过封闭形式的解决方案或自举估计），岭回归，套索和弹性网。在一个特定的模型开发数据集，惩罚方法可能是不可靠的，因为调整参数估计有很大的不确定性。当开发数据集的有效样本量较小且模型的Cox-Snell较低时，这是最令人担忧的。这个问题可能会导致新个体模型预测的相当大的错误校准。惩罚方法并不是一个“空白菜单”;它们并不能保证开发出可靠的预测模型。它们在最需要的时候更不可靠（即，当过拟合可能较大时）。我们建议最好将其应用于大有效样本量，如最近的样本量计算所确定的，旨在最大限度地减少模型过拟合的可能性并精确估计关键参数。在开发临床预测模型时，建议使用惩罚和收缩技术来解决过拟合问题。一些方法论文章建议惩罚方法是一个“全权委托”，并解决任何问题与过度拟合。我们发现惩罚方法可能是不可靠的，因为它们的未知收缩和调整参数估计往往估计有很大的不确定性。虽然平均而言，惩罚方法将改进标准估计方法，但在特定数据集中，它们往往不可靠。最有问题的数据集是那些有效样本量小的数据集，并且开发的模型具有远离1的Cox-Snell，这对于二元和时间事件结果的预测模型是常见的。惩罚方法最好用在有足够大的开发数据集的情况下，如通过样本量计算确定的，以尽量减少模型过度拟合的可能性并精确估计关键参数。当样本量足够大时，可以使用任何研究过的惩罚或收缩方法，因为它们的表现应该类似于并且优于未惩罚回归，除非样本量非常大并且很大。
When developing a clinical prediction model, penalization techniques are recommended to address overfitting, as they shrink predictor effect estimates toward the null and reduce mean-square prediction error in new individuals. However, shrinkage and penalty terms (‘tuning parameters’) are estimated with uncertainty from the development data set. We examined the magnitude of this uncertainty and the subsequent impact on prediction model performance. This study comprises applied examples and a simulation study of the following methods: uniform shrinkage (estimated via a closed-form solution or bootstrapping), ridge regression, the lasso, and elastic net. In a particular model development data set, penalization methods can be unreliable because tuning parameters are estimated with large uncertainty. This is of most concern when development data sets have a small effective sample size and the model's Cox-Snell is low. The problem can lead to considerable miscalibration of model predictions in new individuals. Penalization methods are not a ‘carte blanche’; they do not guarantee a reliable prediction model is developed. They are more unreliable when needed most (i.e., when overfitting may be large). We recommend they are best applied with large effective sample sizes, as identified from recent sample size calculations that aim to minimize the potential for model overfitting and precisely estimate key parameters. When developing a clinical prediction model, penalization and shrinkage techniques are recommended to address overfitting. Some methodology articles suggest penalization methods are a ‘carte blanche’ and resolve any issues to do with overfitting. We show that penalization methods can be unreliable, as their unknown shrinkage and tuning parameter estimates are often estimated with large uncertainty. Although penalization methods will, on average, improve on standard estimation methods, in a particular data set, they are often unreliable. The most problematic data sets are those with small effective sample sizes and where the developed model has a Cox-Snell far from 1, which is common for prediction models of binary and time-to-event outcomes. Penalization methods are best used in situations when a sufficiently large development data set is available, as identified from sample size calculations to minimize the potential for model overfitting and precisely estimate key parameters. When the sample size is adequately large, any of the studied penalization or shrinkage methods can be used, as they should perform similarly and better than unpenalized regression unless sample size is extremely large and is large.
登录
查看更多内容
影响因子：
2.3
作者：
Copas, J B
通讯作者：
Copas, J B
影响因子：
3.6
作者：
Steyerberg, EW;Eijkemans, MJC;Habbema, JDF
通讯作者：
Habbema, JDF
影响因子：
2.5
作者：
HOERL, AE;KENNARD, RW
通讯作者：
KENNARD, RW
影响因子：
1.5
作者：
Steyerberg, EW;Eijkemans, MJC;Habbema, JDF
通讯作者：
Habbema, JDF
影响因子：
2
作者：
Ambler, G.;Seaman, S.;Omar, R. Z.
通讯作者：
Omar, R. Z.