iFixFlakies: a framework for automatically fixing order-dependent flaky tests
iFixFlakies: a framework for automatically fixing order-dependent flaky tests
复制标题
DOI：
10.1145/3338906.3338925
复制
发表时间：
2019-08
期刊：
影响因子：
--
通讯作者：
A. Shi;Wing Lam;Reed Oei;Tao Xie;D. Marinov
中科院分区：
文献类型：
--
作者：
A. Shi;Wing Lam;Reed Oei;Tao Xie;D. Marinov
回归测试提供了重要的通过或失败信号，开发人员在代码更改后用来做出决策。但是，即使代码没有更改，也会通过或失败的片状测试可能会误导开发人员。一种常见的片状测试是订单依赖性测试，它通过或失败，具体取决于运行测试的顺序。固定订单依赖性测试通常是乏味且耗时的。我们提出了Ifixflakies，这是一个自动固定订单依赖性测试的框架。 Ifixflakies中的关键见解是，测试套件通常已经具有测试，我们称之为助手，其逻辑重置或设置了状态以进行订单依赖性测试。 Ifixflakies搜索了使订单依赖性测试的帮助者搜索测试套件，然后使用这些帮助者的代码为订单依赖性测试提供补丁。我们对来自公共数据集的110项真正取决于有序的测试的评估表明，其中有58个具有帮助者，ifixflakies可以修复所有58个。我们为56个订单依赖性测试打开了拉动请求（58个中的2个已经解决），并且开发人员已经接受了。提取其中21个请求，其余所有内容仍在待处理中。
Regression testing provides important pass or fail signals that developers use to make decisions after code changes. However, flaky tests, which pass or fail even when the code has not changed, can mislead developers. A common kind of flaky tests are order-dependent tests, which pass or fail depending on the order in which the tests are run. Fixing order-dependent tests is often tedious and time-consuming. We propose iFixFlakies, a framework for automatically fixing order-dependent tests. The key insight in iFixFlakies is that test suites often already have tests, which we call helpers, whose logic resets or sets the states for order-dependent tests to pass. iFixFlakies searches a test suite for helpers that make the order-dependent tests pass and then recommends patches for the order-dependent tests using code from these helpers. Our evaluation on 110 truly orderdependent tests from a public dataset shows that 58 of them have helpers, and iFixFlakies can fix all 58. We opened pull requests for 56 order-dependent tests (2 of 58 were already fixed), and developers have already accepted pull requests for 21 of them, with all the remaining ones still pending.