An empirical comparison of voting classification algorithms: Bagging, boosting, and variants
An empirical comparison of voting classification algorithms: Bagging, boosting, and variants
复制标题
DOI：
10.1023/a:1007515423169
复制
发表时间：
1999-07-01
期刊：
影响因子：
7.5
通讯作者：
Kohavi, R
中科院分区：
文献类型：
--
作者：
Bauer, E;Kohavi, R
投票分类算法的方法，如Bagging和AdaBoost，已被证明在提高人工和真实世界数据集的某些分类器的准确性方面非常成功。我们回顾这些算法，并描述了一个大型的实证研究比较几个变种结合决策树诱导（三个变种）和朴素贝叶斯诱导。这项研究的目的是提高我们的理解，为什么以及当这些算法，它使用扰动，重新加权，和组合技术，影响分类错误。我们提供了误差的偏差和方差分解，以显示不同的方法和变量如何影响这两个项。这使我们能够确定Bagging减少了不稳定方法的方差，而boosting方法（AdaBoost和Arc-x4）减少了不稳定方法的偏差和方差，但增加了非常稳定的朴素贝叶斯的方差。我们观察到，如果使用重新加权而不是重新加权，Arc-x4的行为与AdaBoost不同，这表明了根本的差异。本文介绍的一些投票变体包括：修剪与不修剪，使用概率估计，权重扰动（摇摆）和数据的后拟合。我们发现，当使用概率估计结合无修剪时，以及当数据进行后拟合时，Bagging会有所改善。我们测量树的大小，并显示出一个有趣的正相关性之间的平均树的大小在AdaBoost试验和它的成功减少错误的增加。我们比较了投票方法和非投票方法的均方误差，结果表明，投票方法可以显著降低均方误差。实际问题中出现的实施升压算法进行了探讨，包括数值不稳定性和下溢。我们使用散点图以图形方式显示AdaBoost如何重新加权实例，不仅强调“硬”区域，还强调离群值和噪声。
Methods for voting classification algorithms, such as Bagging and AdaBoost, have been shown to be very successful in improving the accuracy of certain classifiers for artificial and real-world datasets. We review these algorithms and describe a large empirical study comparing several variants in conjunction with a decision tree inducer (three variants) and a Naive-Bayes inducer. The purpose of the study is to improve our understanding of why and when these algorithms, which use perturbation, reweighting, and combination techniques, affect classification error. We provide a bias and variance decomposition of the error to show how different methods and variants influence these two terms. This allowed us to determine that Bagging reduced variance of unstable methods, while boosting methods (AdaBoost and Arc-x4) reduced both the bias and variance of unstable methods but increased the variance for Naive-Bayes, which was very stable. We observed that Arc-x4 behaves differently than AdaBoost if reweighting is used instead of resampling, indicating a fundamental difference. Voting variants, some of which are introduced in this paper, include: pruning versus no pruning, use of probabilistic estimates, weight perturbations (Wagging), and backfitting of data. We found that Bagging improves when probabilistic estimates in conjunction with no-pruning are used, as well as when the data was backfit. We measure tree sizes and show an interesting positive correlation between the increase in the average tree size in AdaBoost trials and its success in reducing the error. We compare the mean-squared error of voting methods to non-voting methods and show that the voting methods lead to large and significant reductions in the mean-squared errors. Practical problems that arise in implementing boosting algorithms are explored, including numerical instabilities and underflows. We use scatterplots that graphically show how AdaBoost reweights instances, emphasizing not only "hard" areas but also outliers and noise.