Multiple-rule bias in the comparison of classification rules.
Multiple-rule bias in the comparison of classification rules.
复制标题
分类规则比较中的多规则偏差。
DOI：
10.1093/bioinformatics/btr262
复制
发表时间：
2011
期刊：
影响因子：
--
通讯作者：
Dougherty,EdwardR
中科院分区：
文献类型：
--
作者：
Yousefi,MohammadmahdiR;Hua,Jianping;Dougherty,EdwardR
动机：在生物信息学社区中，关于报告结果过于乐观的讨论越来越多。导致分类中过度乐观的两种方法是(i)在数据集上报告所提出的分类规则表现良好的结果，以及（ii）在单个数据集上比较多个分类规则，旨在显示某个规则的优势。结果：本文对第二个问题和“多规则偏差”进行了仔细的概率分析，这是由于在数据集上选择具有最小估计误差的分类规则而导致的。它量化了这种偏差，对应于估计具有最小估计误差的分类规则的期望真实误差，并通过估计数据集上的比较优势来估计所选分类规则相对于其他规则的真实比较优势。使用许多分类规则和误差估计器，将分析应用于合成数据和实际数据。可用性：我们已经用C代码实现了综合数据分布模型、分类规则、特征选择例程和误差估计方法。多规则分析的代码在MATLAB中实现。源代码可从http://gsp.tamu.edu/Publications/supplementary/yousefi11a/获得。补充模拟结果也包括在内。联系：edward@ece.tamu.eduSupplementary信息：补充数据可在bioinformaticsonline上获得。
Motivation:There is growing discussion in the bioinformatics community concerning overoptimism of reported results. Two approaches contributing to overoptimism in classification are (i) the reporting of results on datasets for which a proposed classification rule performs well and (ii) the comparison of multiple classification rules on a single dataset that purports to show the advantage of a certain rule.Results:This article provides a careful probabilistic analysis of the second issue and the ‘multiple-rule bias’, resulting from choosing a classification rule having minimum estimated error on the dataset. It quantifies this bias corresponding to estimating the expected true error of the classification rule possessing minimum estimated error and it characterizes the bias from estimating the true comparative advantage of the chosen classification rule relative to the others by the estimated comparative advantage on the dataset. The analysis is applied to both synthetic and real data using a number of classification rules and error estimators.Availability:We have implemented in C code the synthetic data distribution model, classification rules, feature selection routines and error estimation methods. The code for multiple-rule analysis is implemented in MATLAB. The source code is available at http://gsp.tamu.edu/Publications/supplementary/yousefi11a/. Supplementary simulation results are also included.Contact:edward@ece.tamu.eduSupplementary Information:Supplementary data are available atBioinformaticsonline.