Probabilistic and Systematic Coverage of Consecutive Test-Method Pairs for Detecting Order-Dependent Flaky Tests
Probabilistic and Systematic Coverage of Consecutive Test-Method Pairs for Detecting Order-Dependent Flaky Tests
复制标题
DOI：
10.1007/978-3-030-72016-2_15
复制
发表时间：
2021-03-01
期刊：
影响因子：
--
通讯作者：
Lam W
中科院分区：
文献类型：
--
作者：
Wei A;Yi P;Xie T;Marinov D;Lam W
软件开发人员经常通过针对其代码运行一组测试来检查其代码更改。在同一代码版本上运行时可能会通过或失败的测试称为片状测试。这些测试是一个主要问题，因为当失败与这些更改无关时，他们可能会误导开发人员对最近的代码更改进行调试。片状测试的一个突出类别是订单依赖性（OD）测试，它可以根据运行的测试集的顺序确定性地通过或失败。通过预先检测OD测试，开发人员可以在更改代码之前修复这些测试。由于探索所有可能的订单所需的高成本（n！n测试的n！排列），先前的工作已经开发了随机订单以检测OD测试的工具。实验表明，随机化可以检测到许多OD测试，并且大多数OD测试仅取决于另一个测试失败。但是，没有分析随机订单检测OD检验的概率。在本文中，我们介绍了第一个这样的分析，并提出了对随机测试订单采样以增加概率的简单更改。我们最终提出了一种新颖的算法，以系统地探索所有连续的测试对，以确保检测所有取决于其他测试的OD测试，同时运行的订单和测试要少于简单地运行所有测试对。
Software developers frequently check their code changes by running a set of tests against their code. Tests that can nondeterministically pass or fail when run on the same code version are called flaky tests. These tests are a major problem because they can mislead developers to debug their recent code changes when the failures are unrelated to these changes. One prominent category of flaky tests is order-dependent (OD) tests, which can deterministically pass or fail depending on the order in which the set of tests are run. By detecting OD tests in advance, developers can fix these tests before they change their code. Due to the high cost required to explore all possible orders (n! permutations for n tests), prior work has developed tools that randomize orders to detect OD tests. Experiments have shown that randomization can detect many OD tests, and that most OD tests depend on just one other test to fail. However, there was no analysis of the probability that randomized orders detect OD tests. In this paper, we present the first such analysis and also present a simple change for sampling random test orders to increase the probability. We finally present a novel algorithm to systematically explore all consecutive pairs of tests, guaranteeing to detect all OD tests that depend on one other test, while running substantially fewer orders and tests than simply running all test pairs.