Preempting Flaky Tests via Non-Idempotent-Outcome Tests
Preempting Flaky Tests via Non-Idempotent-Outcome Tests
复制标题
DOI：
10.1145/3510003.3510170
复制
发表时间：
2022-05
期刊：
影响因子：
--
通讯作者：
Anjiang Wei;Pu Yi;Zhengxi Li;Tao Xie;D. Marinov;Wing Lam
中科院分区：
文献类型：
--
作者：
Anjiang Wei;Pu Yi;Zhengxi Li;Tao Xie;D. Marinov;Wing Lam
回归测试对软件开发有很大的帮助，但它可能会被重复测试严重破坏，重复测试在同一代码提交上既可能通过也可能失败，似乎是不确定的。片状测试在研究和工业中都是一个新兴的话题。先前的工作已经确定了多个类别的重复测试，开发了检测这些重复测试的技术，并分析了一些检测到的重复测试。主动检测，即，抢占式，测试，我们建议检测非幂等结果（NIO）的测试，一个新的类别相关的测试。特别是，我们在相同的测试执行环境中运行每个测试两次，例如，在同一个Java虚拟机上运行两次Java测试。如果测试在第一次运行中通过，但在第二次运行中失败，则该测试是NIO。每个NIO测试都有副作用，并且会“自污染”测试运行中共享的状态。我们在Java和Python开源项目上进行了实验，检测了223个NIO Java测试和138个NIO Python测试。我们已经检查了所有361个检测到的测试，并打开了修复268个测试的pull请求，其中192个已经接受，只有6个被拒绝，其余70个待定。
Regression testing can greatly help in software development, but it can be seriously undermined by flaky tests, which can both pass and fail, seemingly nondeterministically, on the same code commit. Flaky tests are an emerging topic in both research and industry. Prior work has identified multiple categories of flaky tests, developed techniques for detecting these flaky tests, and analyzed some detected flaky tests. To proactively detect, i.e., preempt, flaky tests, we propose to detect non-idempotent-outcome (NIO) tests, a novel category related to flaky tests. In particular, we run each test twice in the same test execution environment, e.g., run each Java test twice in the same Java Virtual Machine. A test is NIO if it passes in the first run but fails in the second. Each NIO test has side effects and “self-pollutes” the state shared among test runs. We perform experiments on both Java and Python open-source projects, detecting 223 NIO Java tests and 138 NIO Python tests. We have inspected all 361 detected tests and opened pull requests that fix 268 tests, with 192 already accepted, only 6 rejected, and the remaining 70 pending.