Shake It! Detecting Flaky Tests Caused by Concurrency with Shaker
Shake It! Detecting Flaky Tests Caused by Concurrency with Shaker
复制标题
摇一摇！
DOI：
--
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
Marcelo d’Amorim
中科院分区：
文献类型：
--
作者：
Denini Silva;Leopoldo Teixeira;Marcelo d’Amorim
当测试不确定地通过或失败时，就被认为是不可靠的。测试剥落会对回归测试的有效性产生负面影响，从而影响软件演化。测试剥落的检测是一个重要而又具有挑战性的问题。重新运行是工业中最流行的检测测试剥落的方法。它在固定的代码版本上多次重新执行测试套件，在执行过程中寻找不一致的输出。不幸的是，重新运行成本很高，而且不可靠。本文提出了一种轻量级技术Shaker，以提高重新运行检测薄片测试的能力。Shaker在执行环境中增加了噪音(例如，它增加了压力源任务来竞争CPU或内存)。它建立在以下观察结果的基础上，即并发性是剥落的重要来源，并且在环境中添加噪声会干扰事件的排序，从而影响测试输出。我们在一个包含11个Android应用程序的数据集上进行了实验。结果非常令人鼓舞。Shaker发现了比重新运行(分别占总数的95%和37.5%)多得多的不可靠测试，并且发现这些不可靠测试的速度要快得多。此外，Shaker还发现了61个新的薄片测试，这些测试在重新运行的50次执行中没有被检测到。
A test is said to be flaky when it non-deterministically passes or fails. Test flakiness negatively affects the effectiveness of regression testing and, consequently, impacts software evolution. Detecting test flakiness is an important and challenging problem. ReRun is the most popular approach in industry to detect test flakiness. It re-executes a test suite on a fixed code version multiple times, looking for inconsistent outputs across executions. Unfortunately, ReRun is costly and unreliable. This paper proposes SHAKER, a lightweight technique to improve the ability of ReRun to detect flaky tests. SHAKER adds noise in the execution environment (e.g., it adds stressor tasks to compete for the CPU or memory). It builds on the observations that concurrency is an important source of flakiness and that adding noise in the environment can interfere in the ordering of events and, consequently, influence on the test outputs. We conducted experiments on a data set with 11 Android apps. Results are very encouraging. SHAKER discovered many more flaky tests than ReRun (95% and 37.5% of the total, respectively) and discovered these flaky tests much faster. In addition, SHAKER was able to reveal 61 new flaky tests that went undetected in 50 re-executions with ReRun.
DOI：
10.1145/3338906.3338925
发表时间：
2019-08
期刊：
Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
影响因子：
--
作者：
A. Shi;Wing Lam;Reed Oei;Tao Xie;D. Marinov
通讯作者：
A. Shi;Wing Lam;Reed Oei;Tao Xie;D. Marinov