Prediction error estimation: a comparison of resampling methods
Prediction error estimation: a comparison of resampling methods
复制标题
DOI：
10.1093/bioinformatics/bti499
复制
发表时间：
2005-08-01
期刊：
影响因子：
5.8
通讯作者：
Pfeiffer, RM
中科院分区：
文献类型：
--
作者：
Molinaro, AM;Simon, R;Pfeiffer, RM
动机：在基因组研究中，成千上万的特征是在相对较少的样本上收集的。这些研究的目标之一是建立分类器来预测未来观察的结果。这个过程有三个内在的步骤：特征选择，模型选择和预测评估。随着重点预测评估，我们比较了几种方法估计的“真正的”预测误差的预测模型中存在的feature selection.Results：对于小的研究，功能是从成千上万的候选人中选择，重新替换和简单的分裂样本估计严重偏置。在这些小样本中，留一交叉验证（LOOCV），10倍交叉验证（CV）和.632+ bootstrap对于对角判别分析，最近邻和分类树具有最小的偏差。LOOCV和10倍CV线性判别分析的偏倚最小。此外，LOOCV、5倍和10倍CV以及.632+ bootstrap的均方误差最低。.632+ bootstrap在具有强信噪比的小样本量中非常有偏差。随着可获得的样品数量的增加，不同方法之间的性能差异减小。联系人：annette.molinaro@yale. edu补充信息：Molinaro等人（2005）（http：//linus.nci.nih.gov/brb/TechReport.htm）提供了用于模拟和分析的结果和R代码的完整汇编。
Motivation: In genomic studies, thousands of features are collected on relatively few samples. One of the goals of these studies is to build classifiers to predict the outcome of future observations. There are three inherent steps to this process: feature selection, model selection and prediction assessment. With a focus on prediction assessment, we compare several methods for estimating the 'true' prediction error of a prediction model in the presence of feature selection.Results: For small studies where features are selected from thousands of candidates, the resubstitution and simple split-sample estimates are seriously biased. In these small samples, leave-one-out cross-validation (LOOCV), 10-fold cross-validation (CV) and the .632+ bootstrap have the smallest bias for diagonal discriminant analysis, nearest neighbor and classification trees. LOOCV and 10-fold CV have the smallest bias for linear discriminant analysis. Additionally, LOOCV, 5- and 10-fold CV, and the .632+ bootstrap have the lowest mean square error. The .632+ bootstrap is quite biased in small sample sizes with strong signal-to-noise ratios. Differences in performance among resampling methods are reduced as the number of specimens available increase.Contact: annette.molinaro@yale.eduSupplementary Information: A complete compilation of results and R code for simulations and analyses are available in Molinaro et al. (2005) (http://linus.nci.nih.gov/brb/TechReport.htm).