Initial Results on Counting Test Orders for Order-Dependent Flaky Tests Using Alloy
Initial Results on Counting Test Orders for Order-Dependent Flaky Tests Using Alloy
复制标题
DOI：
10.1007/978-3-031-04673-5_9
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Wenxi Wang;Pu Yi;S. Khurshid;D. Marinov
中科院分区：
文献类型：
--
作者：
Wenxi Wang;Pu Yi;S. Khurshid;D. Marinov
对于同一个被测代码，不稳定的测试可能看起来不确定地通过或失败。不完整的测试对回归测试是有害的，因为在代码更改之前通过而在代码更改之后失败的测试不能可靠地指示代码更改中的问题。顺序相关测试是一个重要的类别，它的通过或失败取决于测试套件中测试的顺序。先前的工作已经考虑了计算通过或失败的测试顺序的问题，给出了测试套件中测试的关系。然而，以前的工作并没有解决这些关系的最一般的情况下。本文展示了如何在Alloy建模语言中对测试顺序计数问题进行编码，以及如何使用命题模型计数器来获得测试顺序的计数。我们举例说明，合金使得它很容易处理，即使是最一般的情况。结果表明，这个问题产生了具有挑战性的命题公式的最先进的模型计数器。
Flaky tests can seemingly nondeterministically pass or fail for the same code under test. Flaky tests are detrimental to regression testing because tests that pass before code changes and fail after code changes do not reliably indicate problems in code changes. An important category of flaky tests is order-dependent tests that pass or fail based on the order of tests in the test suite. Prior work has considered the problem of counting test orders that pass or fail, given relationships of tests within a test suite. However, prior work has not addressed the most general case of these relationships. This paper shows how to encode the problem of counting test orders in the Alloy modeling language and how to use propositional model counters to obtain the count for test orders. We illustrate that Alloy makes it easy to handle even the most general case. The results show that this problem produces challenging propositional formulas for the state-of-the-art model counters.