Test FLARE (Test Flakiness Automated Reproduction and Explanation)

软件故障的成本对全球经济来说是一个巨大的负担，2017年估计至少为1.3万亿英镑。因此，软件测试作为对失败的重要防御，贡献了软件开发工作和成本的很大一部分。不稳定的测试对分配给软件开发的资源来说是一种特别的压力，因为它们在不更改测试或项目代码的情况下断断续续地通过或失败，通常有令人恼火的、不明显的原因。不可靠的测试是从根本上说并不总是告诉真相的测试：它们可能在代码正常工作时失败，而在代码不正常时通过。因为开发人员不能再相信他们的测试结果，他们无法获得软件正确工作的信心，从而潜在地将最终用户暴露在软件故障的后果中。不可靠的测试在工业中是一种常见的现象，它极大地破坏了软件开发——即使对于拥有大量资源来解决这些问题的公司，如微软、Facebook和b谷歌也是如此。一个测试可以产生不同的通过/失败（即，不稳定的）结果，因为它运行的执行环境与它的行为和/或它测试的代码交互的不同的、不可预测的方式。例如，一台机器可能正在经历繁重的并发任务负载，导致它缓慢地执行测试，有时会触发被测代码中的超时，有时则不会。或者，网络访问在测试基础设施上是不稳定的，这意味着网络资源的可用性可能会受到损害。或者，测试逻辑下的程序依赖于时间和日期。这些只是几个真实的例子，说明考试可能会以不同的方式出现问题。对于某些环境条件，测试通过，但在另一个上下文中，相同的测试失败。为了消除不稳定的测试行为，开发人员必须修改测试代码或其测试代码以控制其执行环境的各个方面；即，其间歇性行为的潜在来源。但是，为了准确地评估代码执行行为的差异以及代码中需要更改的位置，开发人员必须能够可靠地重现不同的通过/失败测试结果。然而，这不仅需要重现导致片状行为的环境条件，还需要弄清楚最初导致片状行为的环境条件是什么。对于开发人员来说，手动解决这些问题和重现零散的测试可能极具挑战性，因为相关的环境条件(a)是间歇性的；并且(b)可能与测试实际检查的任何内容无关，并且/或者远离被测试的代码。现有的研究技术不足以解决这些问题，而且尽管开发人员鼓励消除漏洞，例如b谷歌报告说，令人惊讶的是，七分之一的测试是漏洞。Test FLARE项目将做什么：Test FLARE项目将开发并经验性地评估能够(1)自动再现由执行环境引起的不稳定行为的技术。它还将为开发人员提供(2)自动化的、人类可读的解释，帮助开发人员进一步理解不稳定行为的原因。

The cost of software failures is a huge burden to the worldwide economy that was estimated to be at least £1.3 trillion in 2017. Consequently, software testing, a vital defence against failures, contributes to a large proportion of software development effort and cost. Flaky tests are a particular strain on resources allocated to software development, because they intermittently pass and fail without changes to tests or project code, with often maddening, non-obvious causes. Flaky tests are tests that fundamentally do not always tell the truth: they can fail when code is working, and pass when it isn't. Because developers can no longer trust the results of their tests, they are unable to gain confidence that software is working correctly, potentially exposing end-users to the consequences of software failures. Flaky tests are a common occurrence in industry, significantly disrupting software development - even for companies with the greatest amount of resources to tackle them, such as Microsoft, Facebook, and Google.A test can produce different pass/fail (i.e., flaky) outcomes because of differing, unpredicted ways that the execution environment in which it runs interacts with its behaviour and/or the code that it tests. For instance, a machine may be experiencing a heavy concurrent task load, causing it to execute tests slowly, sometimes triggering timeouts in the code under test, and sometimes not. Or, network access is erratic on the testing infrastructure, meaning the availability of network resources may be compromised. Or, a program under test's logic is time and date dependent. These are just a few real examples of the different ways in which tests can be flaky. For some environmental conditions, the test passes, but in an alternative context, the same test fails.To remove flaky test behaviour, a developer has to modify test code or the code that it tests to control for aspects of its execution environment; i.e., the potential sources of its intermittent behaviour. But to accurately assess the differences in code execution behaviour and the places in the code that need to be changed, a developer must be able to reliably reproduce the differing pass/fail test outcomes. However, this not only involves recreating the environmental conditions that lead to the flaky behaviour, but also figuring out exactly what the environmental conditions were that caused the flakiness in the first place. Solving these issues and reproducing flaky tests manually can be extremely challenging for developers since the environmental conditions concerned (a) are intermittent; and (b) may be unrelated to anything the test is actually checking, and/or far-removed from the code being tested. Existing research techniques are insufficient for addressing these problems, and despite developer incentives for removing flakiness, Google, for instance, reports an astonishing one in seven tests as flaky.What the Test FLARE Project Will Do: The Test FLARE project will develop and empirically evaluate techniques capable of (1) automatically reproducing flaky behaviour that is due to the execution environment. It will also provide developers with (2) automated, human-readable explanations that help developers further understand the reasons for the flaky behaviour.