Know You Neighbor: Fast Static Prediction of Test Flakiness
Know You Neighbor: Fast Static Prediction of Test Flakiness
复制标题
DOI：
10.1109/access.2021.3082424
复制
发表时间：
2021
期刊：
影响因子：
3.9
通讯作者：
R. Verdecchia;Emilio Cruciani;Breno Miranda;A. Bertolino
中科院分区：
文献类型：
--
作者：
R. Verdecchia;Emilio Cruciani;Breno Miranda;A. Bertolino
背景：虚假测试拖慢了变更发布速度，浪费了测试时间和精力，从而困扰着持续集成环境中的回归测试。尽管人们对减轻测试缺陷负担的兴趣与日俱增，但如何高效、有效地检测缺陷测试仍是一个悬而未决的问题。研究目的在本研究中，我们介绍并评估了 FLAST，这是一种旨在静态预测测试缺陷的方法。FLAST 利用向量空间建模、相似性搜索、降维和 $k$ - 最近邻分类等方法，及时有效地检测测试缺陷。方法：为了深入了解 FLAST 的效率和有效性，我们对该方法进行了实证评估，考虑了 13 个真实世界的项目，共计 1,383 个缺陷测试和 26,702 个非缺陷测试。我们还考虑了一个平衡数据集，其中包括 1,402 个真实世界的片状测试和同样多的非片状测试，将 FLAST 与最先进的测试片状检测方法进行了定量比较。结果从结果中我们可以看出，FLAST 的有效性与最先进的方法不相上下，同时在效率方面也有相当大的提高。此外，结果还证明了如何通过调整方法的阈值使 FLAST 更加保守，从而减少误报，但代价是漏掉更多潜在的虚假测试。结论收集到的结果表明，FLAST 提供了一种快速、低成本和可靠的方法，可用于指导测试的重新运行，或对新的潜在缺陷测试进行把关。
Context: Flaky tests plague regression testing in Continuous Integration environments by slowing down change releases and wasting testing time and effort. Despite the growing interest in mitigating the burden of test flakiness, how to efficiently and effectively detect flaky tests is still an open problem. Objective: In this study, we present and evaluate FLAST, an approach designed to statically predict test flakiness. FLAST leverages vector-space modeling, similarity search, dimensionality reduction, and $k$ -Nearest Neighbor classification in order to timely and efficiently detect test flakiness. Method: In order to gain insights into the efficiency and effectiveness of FLAST, we conduct an empirical evaluation of the approach by considering 13 real-world projects, for a total of 1,383 flaky and 26,702 non-flaky tests. We carry out a quantitative comparison of FLAST with the state-of-the-art methods to detect test flakiness, by considering a balanced dataset comprising 1,402 real-world flaky and as many non-flaky tests. Results: From the results we observe that the effectiveness of FLAST is comparable with the state-of-the-art, while providing considerable gains in terms of efficiency. In addition, the results demonstrate how by tuning the threshold of the approach FLAST can be made more conservative, so to reduce false positives, at the cost of missing more potentially flaky tests. Conclusion: The collected results demonstrate that FLAST provides a fast, low-cost and reliable approach that can be used to guide test rerunning, or to gate the inclusion of new potentially flaky tests.