DEFECTS - Comparable and Externally Valid Software Defect Prediction

DEFECTS - 可比较且外部有效的软件缺陷预测

• 批准号: 402774445
• 负责人: Professor Dr. Steffen Herbold
• 金额: --
• 依托单位: Lehrstuhl für AI Engineering
• 依托单位国家: 德国
• 项目类别: Research Grants
• 财政年份: 2018
• 资助国家: 德国
• 起止时间: 2017-12-31 至 2021-12-31
• 项目状态: 已结题
• 来源: https://gepris.dfg.de/gepris/projekt/402774445?language=en
• 关键词: DEFECTS; Comparable; Externally; Valid; Software

The comparability and reproducibility of empirical software engineering research is, for the most part, an open problem. This statement holds true for the field of software defect prediction. Current research shows that this leads to actual problems regarding the external validity of defect prediction research. Multiple replications conducted by different groups of researchers led to different findings than prior research. Moreover, problems with the currently used data sets were discovered and it was demonstrated that these problems may change conclusions. Thus, defect prediction research faces a replication crisis if these problems are ignored. Within this project, we plan to create a solid foundation for comparable and externally valid defect prediction research. Our approach rests on three pillars. The first pillar is the quality of the data we use for defect prediction experiments. The current studies on data quality do not cover the impact of mislabeled data. This kind of noise affects not only the creation of defect prediction models, but also their evaluation. We will statistically evaluate the noise in current data sets. Based on our findings, we will improve the state of the art of defect labeling and generate large data set with less noise. The quality of our data will be statistically validated. The collected body will be larger than the available defect prediction data sets and thereby facilitate a better generalizability and external validity of results. The second pillar is the replication of the current state of the art. Since prior replications were already contradictory to the original experiments, we believe that a broader replication effort is necessary. Current replications consider only parts of the state of the art, e.g., classifier impact or cross-project defect prediction. Most of the state of the art still was never replicated and diligently compared to other approaches or naïve baselines. Most experiments only used small data sets, which is a key factor for the problems with external validity. We will conduct a conceptual replication of the state of the art of defect prediction. Through this, we will improve the external validity of the defect prediction state of the art and lay the groundwork for a better external validity of future work. The third pillar are guidelines for defect prediction research. In case we cannot get researchers to avoid anti-patterns that led to bad validity of results, our efforts to combat the replication crisis of defect prediction research will only have a short-term effect. To make our results sustainable, we will work together with the defect prediction community to define guidelines that allow researchers to conduct their defect prediction experiments in such a way that we hopefully never face such problems with replicability again.

在大多数情况下，经验软件工程研究的可比性和可重复性是一个开放的问题。这句话适用于软件缺陷预测领域。目前的研究表明，这导致了缺陷预测研究的外部有效性的实际问题。不同研究小组进行的多次重复实验得出了与之前研究不同的结果。此外，发现了当前使用的数据集的问题，并证明这些问题可能会改变结论。因此，如果忽视这些问题，缺陷预测研究将面临复制危机。在这个项目中，我们计划为可比较的和外部有效的缺陷预测研究创建一个坚实的基础。我们的做法基于三个支柱。第一个支柱是我们用于缺陷预测实验的数据质量。目前对数据质量的研究并没有涵盖错标数据的影响。这种噪声不仅影响缺陷预测模型的建立，而且影响缺陷预测模型的评价。我们将对当前数据集中的噪声进行统计评估。基于我们的发现，我们将改进缺陷标记技术的现状，并生成具有更少噪声的大数据集。我们的数据质量将经过统计验证。收集的主体将比可用的缺陷预测数据集更大，从而有助于更好的推广和结果的外部有效性。第二个支柱是对当前技术状态的复制。由于先前的重复实验已经与最初的实验相矛盾，我们认为有必要进行更广泛的重复实验。当前的复制只考虑技术状态的一部分，例如，分类器影响或跨项目缺陷预测。大多数最新的技术仍然没有被复制，也没有与其他方法或naïve基线进行比较。大多数实验只使用小数据集，这是外部效度问题的关键因素。我们将对缺陷预测技术的现状进行概念性的复制。通过此，我们将提高现有缺陷预测现状的外部有效性，为今后工作更好的外部有效性奠定基础。第三个支柱是缺陷预测研究的指导方针。如果我们不能让研究人员避免导致结果有效性差的反模式，那么我们对抗缺陷预测研究的复制危机的努力将只会产生短期效果。为了使我们的结果可持续，我们将与缺陷预测社区一起定义指导方针，允许研究人员以这样一种方式进行他们的缺陷预测实验，我们希望再也不会遇到这种可复制性问题。