DEFECTS - Comparable and Externally Valid Software Defect Prediction

在大多数情况下，经验软件工程研究的可比性和再现性是一个开放的问题。这种说法适用于软件缺陷预测领域。目前的研究表明，这导致了实际问题的外部有效性的缺陷预测研究。由不同研究小组进行的多次重复实验得出了与先前研究不同的结果。此外，发现了目前使用的数据集的问题，并证明这些问题可能会改变结论。因此，如果忽视这些问题，缺陷预测研究将面临复制危机。在这个项目中，我们计划为可比和外部有效的缺陷预测研究奠定坚实的基础。我们的方法基于三个支柱。第一个支柱是我们用于缺陷预测实验的数据质量。目前关于数据质量的研究没有涵盖错误标记数据的影响。这种噪声不仅影响缺陷预测模型的创建，而且影响其评估。我们将统计评估当前数据集中的噪声。基于我们的研究结果，我们将改进缺陷标记的最新技术，并生成具有更少噪声的大数据集。我们的数据质量将得到统计验证。所收集的主体将大于可用的缺陷预测数据集，从而促进结果的更好的概括性和外部有效性。第二个支柱是复制现有技术，由于先前的复制已经与最初的实验相矛盾，我们认为有必要进行更广泛的复制工作。目前的复制仅考虑了部分现有技术，例如，分类器影响或跨项目缺陷预测。大多数最先进的技术仍然从未被复制，并与其他方法或幼稚的基线进行了比较。大多数实验只使用小数据集，这是外部效度问题的关键因素。我们将对缺陷预测的最新技术进行概念上的复制。通过这一点，我们将提高外部效度的缺陷预测的最先进的状态，并奠定了基础，为今后的工作更好的外部效度。第三个支柱是缺陷预测研究的指导方针。如果我们不能让研究人员避免导致结果有效性不佳的反模式，那么我们对抗缺陷预测研究的复制危机的努力只会产生短期效果。为了使我们的结果可持续发展，我们将与缺陷预测社区合作，制定指导方针，允许研究人员以这样一种方式进行缺陷预测实验，希望我们永远不会再面临类似的可复制性问题。

The comparability and reproducibility of empirical software engineering research is, for the most part, an open problem. This statement holds true for the field of software defect prediction. Current research shows that this leads to actual problems regarding the external validity of defect prediction research. Multiple replications conducted by different groups of researchers led to different findings than prior research. Moreover, problems with the currently used data sets were discovered and it was demonstrated that these problems may change conclusions. Thus, defect prediction research faces a replication crisis if these problems are ignored. Within this project, we plan to create a solid foundation for comparable and externally valid defect prediction research. Our approach rests on three pillars. The first pillar is the quality of the data we use for defect prediction experiments. The current studies on data quality do not cover the impact of mislabeled data. This kind of noise affects not only the creation of defect prediction models, but also their evaluation. We will statistically evaluate the noise in current data sets. Based on our findings, we will improve the state of the art of defect labeling and generate large data set with less noise. The quality of our data will be statistically validated. The collected body will be larger than the available defect prediction data sets and thereby facilitate a better generalizability and external validity of results. The second pillar is the replication of the current state of the art. Since prior replications were already contradictory to the original experiments, we believe that a broader replication effort is necessary. Current replications consider only parts of the state of the art, e.g., classifier impact or cross-project defect prediction. Most of the state of the art still was never replicated and diligently compared to other approaches or naïve baselines. Most experiments only used small data sets, which is a key factor for the problems with external validity. We will conduct a conceptual replication of the state of the art of defect prediction. Through this, we will improve the external validity of the defect prediction state of the art and lay the groundwork for a better external validity of future work. The third pillar are guidelines for defect prediction research. In case we cannot get researchers to avoid anti-patterns that led to bad validity of results, our efforts to combat the replication crisis of defect prediction research will only have a short-term effect. To make our results sustainable, we will work together with the defect prediction community to define guidelines that allow researchers to conduct their defect prediction experiments in such a way that we hopefully never face such problems with replicability again.