DEFECTS - Comparable and Externally Valid Software Defect Prediction

经验软件工程研究的可比性和重复性在很大程度上是一个悬而未决的问题。这句话适用于软件缺陷预测领域。目前的研究表明，这导致了关于缺陷预测研究的外部有效性的实际问题。不同研究小组进行的多次重复研究得出了与先前研究不同的结果。此外，还发现了当前使用的数据集存在的问题，并证明这些问题可能会改变结论。因此，如果忽视这些问题，缺陷预测研究将面临复制危机。在这个项目中，我们计划为可比较和外部有效的缺陷预测研究创造一个坚实的基础。我们的方法基于三个支柱。第一个支柱是我们用于缺陷预测实验的数据的质量。目前关于数据质量的研究没有涵盖错误标记的数据的影响。这种噪声不仅影响缺陷预测模型的建立，而且影响其评估。我们将对当前数据集中的噪声进行统计评估。基于我们的发现，我们将改进缺陷标记的技术水平，并以更少的噪声生成大数据集。我们的数据质量将得到统计验证。收集的主体将大于可用的缺陷预测数据集，从而有助于结果的更好的概括性和外部有效性。第二个支柱是复制当前的技术水平。由于之前的复制已经与原始实验相矛盾，我们认为有必要进行更广泛的复制努力。目前的复制只考虑部分现有技术，例如分类器影响或跨项目缺陷预测。与其他方法或天真的基线相比，大多数最先进的技术仍然从未被复制和勤奋地复制。大多数实验只使用较小的数据集，这是外部效度问题的一个关键因素。我们将对缺陷预测的最新技术进行概念性复制。通过这一点，我们将提高缺陷预测的外部效度现状，为更好的未来工作的外部效度奠定基础。第三个支柱是缺陷预测研究的指导方针。如果我们不能让研究人员避免导致结果有效性不佳的反模式，我们对抗缺陷预测研究的复制危机的努力只会产生短期效果。为了使我们的结果可持续，我们将与缺陷预测社区合作，定义指导方针，允许研究人员以这样一种方式进行他们的缺陷预测实验，希望我们永远不会再面临这样的可复制性问题。

The comparability and reproducibility of empirical software engineering research is, for the most part, an open problem. This statement holds true for the field of software defect prediction. Current research shows that this leads to actual problems regarding the external validity of defect prediction research. Multiple replications conducted by different groups of researchers led to different findings than prior research. Moreover, problems with the currently used data sets were discovered and it was demonstrated that these problems may change conclusions. Thus, defect prediction research faces a replication crisis if these problems are ignored. Within this project, we plan to create a solid foundation for comparable and externally valid defect prediction research. Our approach rests on three pillars. The first pillar is the quality of the data we use for defect prediction experiments. The current studies on data quality do not cover the impact of mislabeled data. This kind of noise affects not only the creation of defect prediction models, but also their evaluation. We will statistically evaluate the noise in current data sets. Based on our findings, we will improve the state of the art of defect labeling and generate large data set with less noise. The quality of our data will be statistically validated. The collected body will be larger than the available defect prediction data sets and thereby facilitate a better generalizability and external validity of results. The second pillar is the replication of the current state of the art. Since prior replications were already contradictory to the original experiments, we believe that a broader replication effort is necessary. Current replications consider only parts of the state of the art, e.g., classifier impact or cross-project defect prediction. Most of the state of the art still was never replicated and diligently compared to other approaches or naïve baselines. Most experiments only used small data sets, which is a key factor for the problems with external validity. We will conduct a conceptual replication of the state of the art of defect prediction. Through this, we will improve the external validity of the defect prediction state of the art and lay the groundwork for a better external validity of future work. The third pillar are guidelines for defect prediction research. In case we cannot get researchers to avoid anti-patterns that led to bad validity of results, our efforts to combat the replication crisis of defect prediction research will only have a short-term effect. To make our results sustainable, we will work together with the defect prediction community to define guidelines that allow researchers to conduct their defect prediction experiments in such a way that we hopefully never face such problems with replicability again.