Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints.
Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints.
复制标题
DOI：
10.1186/1471-2288-14-137
复制
发表时间：
2014-12-22
影响因子：
4
通讯作者：
Steyerberg EW
中科院分区：
文献类型：
--
作者：
van der Ploeg T;Austin PC;Steyerberg EW
现代建模技术可能会比经典技术对二元结果提供更准确的预测。我们的目的是研究不同建模技术相对于有效样本量(“数据饥饿度”)的预测性能。我们基于三个临床队列进行了模拟研究：1282例头颈癌患者(5年生存率46.9%)，1731例创伤性脑损伤患者(6个月死亡率22.3%)和3181例轻型颅脑损伤患者(7.6%CT扫描异常)。我们比较了三种相对现代的建模技术：支持向量机(SVM)、神经网络(NN)和随机森林(RF)，以及两种经典技术：Logistic回归(LR)和分类回归树(CART)。我们创建了三个大型人工数据库，分别对受试者进行20倍、10倍和6倍的复制，根据不同的潜在模型生成两种结果。我们将每种建模技术应用于越来越大的开发部分(100次重复)。ROC曲线下面积(AUC)表示每个模型在开发部分和独立验证部分的表现。数据饥饿感由AUC的平台期和微小的乐观情绪(平均表观AUC和验证AUC<；0.01的平均值之间的差异)来定义。我们发现，在每个变量大约20到50个事件时，LR达到了稳定的AUC，其次是CART、支持向量机、神经网络和RF模型。随着样本量的增加和技术等级的相同，乐观程度也会降低。RF、支持向量机和神经网络模型显示出不稳定和高度乐观，即使每个变量有200个事件。与经典建模技术(如LR)相比，现代建模技术(如支持向量机、神经网络和RF)可能需要每个变量10倍以上的事件才能获得稳定的AUC和较小的乐观。这意味着，只有在有非常大的数据集的情况下，这种现代技术才应该用于医学预测问题。本文的在线版本(DOI：10.1186/1471-2288-14-137)包含补充材料，可供授权用户使用。
Modern modelling techniques may potentially provide more accurate predictions of binary outcomes than classical techniques. We aimed to study the predictive performance of different modelling techniques in relation to the effective sample size (“data hungriness”). We performed simulation studies based on three clinical cohorts: 1282 patients with head and neck cancer (with 46.9% 5 year survival), 1731 patients with traumatic brain injury (22.3% 6 month mortality) and 3181 patients with minor head injury (7.6% with CT scan abnormalities). We compared three relatively modern modelling techniques: support vector machines (SVM), neural nets (NN), and random forests (RF) and two classical techniques: logistic regression (LR) and classification and regression trees (CART). We created three large artificial databases with 20 fold, 10 fold and 6 fold replication of subjects, where we generated dichotomous outcomes according to different underlying models. We applied each modelling technique to increasingly larger development parts (100 repetitions). The area under the ROC-curve (AUC) indicated the performance of each model in the development part and in an independent validation part. Data hungriness was defined by plateauing of AUC and small optimism (difference between the mean apparent AUC and the mean validated AUC <0.01). We found that a stable AUC was reached by LR at approximately 20 to 50 events per variable, followed by CART, SVM, NN and RF models. Optimism decreased with increasing sample sizes and the same ranking of techniques. The RF, SVM and NN models showed instability and a high optimism even with >200 events per variable. Modern modelling techniques such as SVM, NN and RF may need over 10 times as many events per variable to achieve a stable AUC and a small optimism than classical modelling techniques such as LR. This implies that such modern techniques should only be used in medical prediction problems if very large data sets are available. The online version of this article (doi:10.1186/1471-2288-14-137) contains supplementary material, which is available to authorized users.
登录
查看更多内容
影响因子：
4.2
作者：
Marmarou, Anthony;Lu, Juan;Maas, Andrew I. R.
通讯作者：
Maas, Andrew I. R.
影响因子：
1.7
作者：
Austin, Peter C.;Lee, Douglas S.;Tu, Jack V.
通讯作者：
Tu, Jack V.
影响因子：
7.2
作者：
Steyerberg, EW;Harrell, FE;Habbema, JDF
通讯作者：
Habbema, JDF
影响因子：
3.7
作者：
van der Ploeg T;Datema F;Baatenburg de Jong R;Steyerberg EW
通讯作者：
Steyerberg EW
影响因子：
4
作者：
van der Ploeg T;Smits M;Dippel DW;Hunink M;Steyerberg EW
通讯作者：
Steyerberg EW