Cross-study validation for the assessment of prediction algorithms.
Cross-study validation for the assessment of prediction algorithms.
复制标题
DOI：
10.1093/bioinformatics/btu279
复制
发表时间：
2014-06-15
期刊：
影响因子：
--
通讯作者：
Trippa L
中科院分区：
文献类型：
--
作者：
Bernau C;Riester M;Boulesteix AL;Parmigiani G;Huttenhower C;Waldron L;Trippa L
动机：在统计和机器学习文献中已经开发了许多用于高维环境中预测的竞争算法。学习算法及其生成的预测模型通常基于一些示例性数据集中的交叉验证误差估计进行评估。然而，在大多数应用中，预测建模的最终目标是为在不同设置中获得的独立样本提供准确的预测。示例性数据集内的交叉验证可能无法充分反映更广泛应用环境中的性能。研究方法：我们开发并实施了一种系统的方法来“交叉研究验证”，以取代或补充传统的交叉验证时，评估独立数据集的高维预测模型。我们通过模拟和八个雌激素受体阳性乳腺癌微阵列基因表达数据集的集合来说明它，其目标是预测无远处转移生存率（DMFS）。我们计算了训练和验证数据集的所有成对组合的C指数。我们评估了几种替代方案，总结成对验证统计量，并将其与传统的交叉验证进行比较。结果如下：我们的数据驱动的模拟和我们的应用与8个乳腺癌微阵列数据集的生存预测，表明标准的交叉验证产生膨胀的歧视精度为所有考虑的算法，当比较交叉研究验证。此外，学习算法的排名不同，这表明在交叉验证中表现最好的算法在通过独立验证进行评估时可能是次优的。可用性：survHD：在高维度中生存包（http：//www.bitbucket.org/lwaldron/survhd）将通过Bioconductor提供。联系方式：沃尔德龙@ hunter.cuny.edu补充信息：补充数据可在生物信息学在线获得。
Motivation: Numerous competing algorithms for prediction in high-dimensional settings have been developed in the statistical and machine-learning literature. Learning algorithms and the prediction models they generate are typically evaluated on the basis of cross-validation error estimates in a few exemplary datasets. However, in most applications, the ultimate goal of prediction modeling is to provide accurate predictions for independent samples obtained in different settings. Cross-validation within exemplary datasets may not adequately reflect performance in the broader application context. Methods: We develop and implement a systematic approach to ‘cross-study validation’, to replace or supplement conventional cross-validation when evaluating high-dimensional prediction models in independent datasets. We illustrate it via simulations and in a collection of eight estrogen-receptor positive breast cancer microarray gene-expression datasets, where the objective is predicting distant metastasis-free survival (DMFS). We computed the C-index for all pairwise combinations of training and validation datasets. We evaluate several alternatives for summarizing the pairwise validation statistics, and compare these to conventional cross-validation. Results: Our data-driven simulations and our application to survival prediction with eight breast cancer microarray datasets, suggest that standard cross-validation produces inflated discrimination accuracy for all algorithms considered, when compared to cross-study validation. Furthermore, the ranking of learning algorithms differs, suggesting that algorithms performing best in cross-validation may be suboptimal when evaluated through independent validation. Availability: The survHD: Survival in High Dimensions package (http://www.bitbucket.org/lwaldron/survhd) will be made available through Bioconductor. Contact: levi.waldron@hunter.cuny.edu Supplementary information: Supplementary data are available at Bioinformatics online.
登录
查看更多内容
影响因子：
12.3
作者：
Gentleman RC;Carey VJ;Bates DM;Bolstad B;Dettling M;Dudoit S;Ellis B;Gautier L;Ge Y;Gentry J;Hornik K;Hothorn T;Huber W;Iacus S;Irizarry R;Leisch F;Li C;Maechler M;Rossini AJ;Sawitzki G;Smith C;Smyth G;Tierney L;Yang JY;Zhang J
通讯作者：
Zhang J
影响因子：
10.3
作者：
Riester, Markus;Wei, Wei;Birrer, Michael J.
通讯作者：
Birrer, Michael J.
影响因子：
1.9
作者：
HARTLEY, HO;SIELKEN, RL
通讯作者：
SIELKEN, RL
影响因子：
5.8
作者：
Molinaro, AM;Simon, R;Pfeiffer, RM
通讯作者：
Pfeiffer, RM
影响因子：
9.5
作者：
Castaldi, Peter J.;Dahabreh, Issa J.;Ioannidis, John P. A.
通讯作者：
Ioannidis, John P. A.