A Comprehensive Investigation of the Role of Imbalanced Learning for Software Defect Prediction
A Comprehensive Investigation of the Role of Imbalanced Learning for Software Defect Prediction
复制标题
DOI：
10.1109/tse.2018.2836442
复制
发表时间：
2019-12
影响因子：
7.4
通讯作者：
Qinbao Song;Yuchen Guo;M. Shepperd
中科院分区：
文献类型：
--
作者：
Qinbao Song;Yuchen Guo;M. Shepperd
上下文：软件缺陷预测（SDP）是软件工程领域的重要挑战，因此已经通过使用机器学习算法进行了许多研究工作。 - 吸引力是这些方法不平衡的常见，旨在解决这个问题。不幸的是，目标不一致：我们进行了全面的实验来探索（a）此问题的基本特征； ）不平衡的学习方法：我们系统地评估27个数据集，7种分类器，7种输入指标和17种不平衡的学习方法（包括任何事情）一个实验设计，可以探索这些因素与单个不平衡学习算法之间的相互作用。 F1和AUC措施）。不平衡（不平衡比率$ 10;中位数= 3.94）（b）除低水平的不平衡是否会损害SDP传统学习的表现；结果始终是（d）分类器的类型，对分类性能的改善，然后是不平衡的学习方法$ {\ sim} 52 \％$约52％的学习者和分类器的组合具有显着的积极效果：本文提供两个实用的指南。套装。不平衡学习的应用可能是有害的。
Context: Software defect prediction (SDP) is an important challenge in the field of software engineering, hence much research work has been conducted, most notably through the use of machine learning algorithms. However, class-imbalance typified by few defective components and many non-defective ones is a common occurrence causing difficulties for these methods. Imbalanced learning aims to deal with this problem and has recently been deployed by some researchers, unfortunately with inconsistent results. Objective: We conduct a comprehensive experiment to explore (a) the basic characteristics of this problem; (b) the effect of imbalanced learning and its interactions with (i) data imbalance, (ii) type of classifier, (iii) input metrics and (iv) imbalanced learning method. Method: We systematically evaluate 27 data sets, 7 classifiers, 7 types of input metrics and 17 imbalanced learning methods (including doing nothing) using an experimental design that enables exploration of interactions between these factors and individual imbalanced learning algorithms. This yields 27 × 7 × 7 × 17 = 22491 results. The Matthews correlation coefficient (MCC) is used as an unbiased performance measure (unlike the more widely used F1 and AUC measures). Results: (a) we found a large majority (87 percent) of 106 public domain data sets exhibit moderate or low level of imbalance (imbalance ratio $10; median = 3.94); (b) anything other than low levels of imbalance clearly harm the performance of traditional learning for SDP; (c) imbalanced learning is more effective on the data sets with moderate or higher imbalance, however negative results are always possible; (d) type of classifier has most impact on the improvement in classification performance followed by the imbalanced learning method itself. Type of input metrics is not influential. (e) only ${\sim} 52\%$∼52% of the combinations of Imbalanced Learner and Classifier have a significant positive effect. Conclusion: This paper offers two practical guidelines. First, imbalanced learning should only be considered for moderate or highly imbalanced SDP data sets. Second, the appropriate combination of imbalanced method and classifier needs to be carefully chosen to ameliorate the imbalanced learning problem for SDP. In contrast, the indiscriminate application of imbalanced learning can be harmful.