CAREER: Mitigating Flaky Tests

该奖项全部或部分由《2021年美国救援计划法案》（公法117-2）资助。软件对日常生活的影响越来越大，这意味着需要高质量、正确的软件。检查软件正确性的最常用方法是通过软件测试。隐含的假设是，测试失败表明存在错误。然而，这种基本假设由于片状测试的存在而失效，片状测试是在同一代码上运行时不确定地通过或失败的测试。直观地说，在被测试的代码或环境中存在预期或甚至期望的不确定性组件（因此不是软件中的真正错误），但是开发人员可能编写了不考虑这种不确定性的不可靠的测试，从而导致测试有时通过，有时失败。当测试套件中存在不稳定的测试时，开发人员就不能再信任他们的测试结果了。如果测试失败不再保证存在真正的bug，那么开发人员可能会选择忽略所有的测试结果，而不是进一步调查。在这一点上，测试变得毫无用处，开发人员冒着严重错误进入发布软件的风险。不可靠的测试是工业界和开源社区的开发人员面临的一个常见问题。这个项目的目标是减轻片状测试的负面影响。为了实现这个目标，这个项目将(1)开发技术，通过利用代码进化来更有效地检测不可靠的测试，(2)开发专门的测试环境，可以减少不可靠的测试失败，而不需要修改测试下的代码，以及(3)增强依赖于测试的软件工程技术，使其能够意识到不可靠的测试失败，从而不被不可靠的测试失败所误导。除了传统的不稳定测试之外，该项目还将研究和开发技术，以检测和修复与非功能属性（如执行时间）相关的不稳定测试。最后，考虑到片状测试与计算再现性困难之间的相似之处，本项目还将(5)开发基于片状测试工作见解的技术，以提高计算研究结果的再现性。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2). Software’s increasing impact on daily life means there is a need high quality, correct software. The most common approach to check for software correctness is through software testing. The implicit assumption is that a failing test signals the presence of a bug. However, this fundamental assumption is invalidated by the presence of flaky tests, which are tests that nondeterministically pass or fail when run on the same code. Intuitively, there is a nondeterministic component in the code under test or environment that is expected or even desired (and therefore not a true bug in the software), but a developer may have written a flaky test that does not take this nondeterminism into account, resulting in the test to sometimes pass and sometimes fail. When there are flaky tests in the test suite, a developer can no longer trust their test results. If the test failures no longer guarantee the presence of a true bug, developers may then choose to ignore all test outcomes instead of investigating further. At this point, testing become effectively useless, and developers risk critical bugs slipping into the released software. Flaky tests are a common problem faced by developers in industry and the open-source community. The goal of this project is to mitigate the negative effects of flaky tests. To accomplish this goal, this project will (1) develop techniques to more efficiently detect flaky tests by leveraging code evolution, (2) develop specialized testing environments that can reduce flaky test failures without needing to modify code under test, and (3) enhance software engineering techniques that rely on testing to become flaky-test aware so as to not be misled by unreliable test failures. Beyond traditional flaky tests, this project will also (4) investigate and develop techniques to detect and repair tests that are flaky with respect to nonfunctional properties, such as execution time. Finally, given the parallels between flaky tests and difficulties in computational reproducibility, this project will also (5) develop techniques to increase the reproducibility of computational research results based on insights from the work on flaky tests.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.