Bootstrapping the out-of-sample predictions for efficient and accurate cross-validation.
Bootstrapping the out-of-sample predictions for efficient and accurate cross-validation.
复制标题
DOI：
10.1007/s10994-018-5714-4
复制
发表时间：
2018
期刊：
影响因子：
7.5
通讯作者：
Borboudakis G
中科院分区：
文献类型：
--
作者：
Tsamardinos I;Greasidou E;Borboudakis G
交叉验证（CV）和样本外性能估计协议通常用于（a）选择用于产生最终预测模型的算法和超参数值（称为配置）的最佳组合，以及（B）估计最终模型的预测性能。然而，交叉验证的最佳配置的性能是乐观的偏见。我们提出了一种有效的引导方法，纠正的偏见，称为引导偏差校正CV（BBC-CV）。BBC-CV的主要思想是引导选择每个配置的样本外预测的最佳性能配置的整个过程，而无需额外的模型训练。与替代方法相比，即嵌套交叉验证（Varma和Simon in BMC Bioinform 7（1）：91，）和Tibshirani和Tibshirani的方法（Ann Appl Stat 822-829，），BBC-CV在计算上更有效，具有更小的方差和偏倚，适用于任何性能指标（准确度，AUC，一致性指数，均方误差）。随后，我们再次采用引导样本外预测的想法来加速CV过程。具体来说，使用基于Bootstrap的统计标准，我们停止在劣质（高概率）配置的新折叠上训练模型。我们命名的方法Bootstrap偏差校正与下降CV（BBCD-CV），这是既有效，并提供准确的性能估计。
Cross-Validation (CV), and out-of-sample performance-estimation protocols in general, are often employed both for (a) selecting the optimal combination of algorithms and values of hyper-parameters (called a configuration) for producing the final predictive model, and (b) estimating the predictive performance of the final model. However, the cross-validated performance of the best configuration is optimistically biased. We present an efficient bootstrap method that corrects for the bias, called Bootstrap Bias Corrected CV (BBC-CV). BBC-CV’s main idea is to bootstrap the whole process of selecting the best-performing configuration on the out-of-sample predictions of each configuration, without additional training of models. In comparison to the alternatives, namely the nested cross-validation (Varma and Simon in BMC Bioinform 7(1):91,) and a method by Tibshirani and Tibshirani (Ann Appl Stat 822–829,), BBC-CV is computationally more efficient, has smaller variance and bias, and is applicable to any metric of performance (accuracy, AUC, concordance index, mean squared error). Subsequently, we employ again the idea of bootstrapping the out-of-sample predictions to speed up the CV process. Specifically, using a bootstrap-based statistical criterion we stop training of models on new folds of inferior (with high probability) configurations. We name the method Bootstrap Bias Corrected with Dropping CV (BBCD-CV) that is both efficient and provides accurate performance estimates.
登录
查看更多内容
影响因子：
5.8
作者：
Lagani, Vincenzo;Athineou, Giorgos;Tsamardinos, Ioannis
通讯作者：
Tsamardinos, Ioannis
影响因子：
2.7
作者：
COCHRAN, WG
通讯作者：
COCHRAN, WG
影响因子：
6.8
作者：
AKAIKE, H
通讯作者：
AKAIKE, H
影响因子：
7.5
作者：
CORTES, C;VAPNIK, V
通讯作者：
VAPNIK, V
影响因子：
168.9
作者：
Iizuka, N;Oka, M;Hamamoto, Y
通讯作者：
Hamamoto, Y