Mining Software Test Histories to Identify Flaky Tests and Model Commit Failure Risk

我的研究从倾听软件开发人员的日常斗争开始，最终为常见的软件测试问题提供基于理论的通用解决方案。 目标1，不稳定的测试：不稳定的软件测试在没有发现错误的情况下失败，浪费开发人员的时间，降低测试结果的可信度。不完整的测试会影响所有项目，尤其是在大型项目或复杂的测试环境中。传统的做法是重新运行测试，或者隔离它们并要求开发人员修复它们。然而，我观察到，重复测试具有不同程度的片状性。例如，在100次运行中有1次失败的测试比在100次运行中有10次失败的测试更可靠。通过量化历史FlakeRate（例如10/100），我们可以使用统计测试来确定测试何时在正常FlakeRate（10%）下失败，或者FlakeRate何时发生变化，表明不稳定和潜在的产品故障。 我的目标是量化先前运行中的测试片状程度，并在统计上识别FlakeRate的变化，作为测试变得不稳定的指示。工具支持将允许开发人员通过FlakeRate不稳定性对测试进行排名。这应该减少调查工作，因为在FlakeRate中没有统计学显著对应变化的重复测试失败是稳定的，不太可能需要调查。 目标2，提交失败风险：一个常见的开发实践是单独测试每个提交，以立即隔离导致测试失败的提交。由于大多数测试不会失败，而且大多数提交都通过了所有测试，因此我们建议根据提交包含失败测试的可能性来批量提交。一个包含10个提交的批处理将需要运行1次测试套件，从而在测试全部通过的场景中节省9次执行。然而，在失败时，二分过程将需要额外的运行。我们将调整提交风险模型，以识别引入bug的更改，以选择包含在大批量中的低风险提交和应单独测试的高风险提交。这些统计方法的适应和发展是复杂的，在生产CI环境中对其进行评估需要系统的严谨性。 目标3，推广和工具：从长远来看，我将通过模拟和演示在广泛的软件环境中提高的效率，增加软件测试人员和开发人员对软件分析的使用。根据我的经验，开发人员只关注减少错误和按时发布软件。这种关注限制了挖掘和分析在实践中的使用。通过关注开发人员的问题，我已经能够并将继续为耗时、不愉快和昂贵的开发问题提供通用的解决方案。

My research begins by listening to the daily struggles of software developers and ultimately provides theoretically based general solutions to common software testing problems. Objective 1, Flaky tests: Flaky software tests fail without identifying a fault and waste developer time and reduce confidence in test outcomes. Flaky tests affect all projects and are particularly damaging on large projects or in complex test environments. The conventional practice is to re-run flaky tests or to quarantine them and require developers to fix them. However, I have observed that flaky tests have varying degrees of flakiness. For example, a test that has a flaky failure 1 in 100 runs is more reliable than a test that has 10 flaky failures 100 runs. By quantifying the historical FlakeRate, e.g. 10/100, we can use statistical tests to determine when a test is failing at its normal FlakeRate, 10%, or when it FlakeRate has changed, indicating instability and a potential product fault. My objective is to quantify the degree of test flakiness from prior runs and statistically identify changes in the FlakeRate as indications that a test has become unstable. Tool support will allow developers to rank tests by FlakeRate instability. This should reduce investigation effort because flaky failures that do not have a statistically significant corresponding change in FlakeRate are stable and less likely to need investigation. Objective 2, Commit failure risk: A common development practice is to test each commit individually to isolate immediately the commit that caused a test failure. Since most tests do not fail and most commits pass all tests, we propose to batch commits based on how likely a commit is to contain a failing test. A batch that contains 10 commits, will require 1 run of the test suite saving 9 executions over the test-all-passing scenario. However, on failure, a bisection process will require additional runs. We will adapt commit risk models that identify bug-introducing changes to the context of selecting low risk commits to be included in large batches and high risk commits that should be tested individually. The adaption and development of these statistical approaches is complex and evaluating them in production CI environments requires systematic rigor. Objective 3, Generalize and Tools: In the long-term, I will increase the use of software analytics by software testers and developers by simulating and demonstrating improved efficiency in a wide range of software environments. In my experience developers have a single focus of reducing faults and releasing software on time. This focus has limited the use of mining and analytics in practice. By focusing on developer problems, I have been able to and will continue to provide general solutions to time consuming, unpleasant, and expensive development problems.