Studying the impact of CI on pull request delivery time in open source projects—a conceptual replication
Studying the impact of CI on pull request delivery time in open source projects—a conceptual replication
复制标题
研究 CI 对开源项目中拉取请求交付时间的影响——概念复制
作者：
Yu;P. Leitner
如今，持续集成（CI）在软件开发过程中不可或缺。采用 CI 的一个核心承诺是可以更快地交付新功能或错误修复。 Bernardo、da Costa 和 Kulesza（2018）最近进行的一项存储库挖掘研究发现，在接受调查的开源项目中，只有大约一半在采用 CI 后实际上更快地交付拉取请求 (PR)，且效果较小。然而，人们对 Bernardo 等人使用的方法存在一些担忧，这可能会限制这一发现的可信度。特别是，它们没有明确控制项目生命周期内拉取请求交付时间的正常变化（独立于 CI 引入）。因此，在我们的工作中，我们对这项研究进行了概念性的复制。第一步，我们使用相同的主题和方法复制他们的研究结果。第二步，我们使用适应的方法来解决相同的核心研究问题。我们使用不同的统计方法（断点回归设计，RDD），该方法对于项目的混杂因素更加稳健，随着时间的推移，项目可能会更快地自然地交付 PR，并且我们引入了一组从未应用 CI 的可比较项目的对照组。最后，我们还评估了使用相同方法抽样的一组新开源项目的原始发现的普遍性。我们发现 Bernardo 等人的研究结果。很大程度上保留在我们的复制中。使用 RDD，我们没有找到强有力的证据表明项目在没有 CI 的情况下交付 PR 的速度更快，而且我们同样没有看到从未引入 CI 的对照组的速度有所提高。此外，从一组新挖掘的项目中获得的结果与原始发现具有可比性。总之，我们认为复制成功。
Nowadays, continuous integration (CI) is indispensable in the software development process. A central promise of adopting CI is that new features or bug fixes can be delivered more quickly. A recent repository mining study by Bernardo, da Costa & Kulesza (2018) found that only about half of the investigated open source projects actually deliver pull requests (PR) faster after adopting CI, with small effect sizes. However, there are some concerns regarding the methodology used by Bernardo et al., which may potentially limit the trustworthiness of this finding. Particularly, they do not explicitly control for normal changes in the pull request delivery time during a project’s lifetime (independently of CI introduction). Hence, in our work, we conduct a conceptual replication of this study. In a first step, we replicate their study results using the same subjects and methodology. In a second step, we address the same core research question using an adapted methodology. We use a different statistical method (regression discontinuity design, RDD) that is more robust towards the confounding factor of projects potentially getting faster in delivering PRs over time naturally, and we introduce a control group of comparable projects that never applied CI. Finally, we also evaluate the generalizability of the original findings on a set of new open source projects sampled using the same methodology. We find that the results of the study by Bernardo et al. largely hold in our replication. Using RDD, we do not find robust evidence of projects getting faster at delivering PRs without CI, and we similarly do not see a speed-up in our control group that never introduced CI. Further, results obtained from a newly mined set of projects are comparable to the original findings. In conclusion, we consider the replication successful.