CAREER: Mitigating Flaky Tests

职业：减少不稳定的测试

• 批准号: 2145774
• 负责人: August Shi
• 金额: $ 59.02万
• 依托单位: University of Texas at Austin
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-07-01 至 2027-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2145774&HistoricalAwards=false
• 关键词: CAREER; Mitigating; Flaky; Tests

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2). Software’s increasing impact on daily life means there is a need high quality, correct software. The most common approach to check for software correctness is through software testing. The implicit assumption is that a failing test signals the presence of a bug. However, this fundamental assumption is invalidated by the presence of flaky tests, which are tests that nondeterministically pass or fail when run on the same code. Intuitively, there is a nondeterministic component in the code under test or environment that is expected or even desired (and therefore not a true bug in the software), but a developer may have written a flaky test that does not take this nondeterminism into account, resulting in the test to sometimes pass and sometimes fail. When there are flaky tests in the test suite, a developer can no longer trust their test results. If the test failures no longer guarantee the presence of a true bug, developers may then choose to ignore all test outcomes instead of investigating further. At this point, testing become effectively useless, and developers risk critical bugs slipping into the released software. Flaky tests are a common problem faced by developers in industry and the open-source community. The goal of this project is to mitigate the negative effects of flaky tests. To accomplish this goal, this project will (1) develop techniques to more efficiently detect flaky tests by leveraging code evolution, (2) develop specialized testing environments that can reduce flaky test failures without needing to modify code under test, and (3) enhance software engineering techniques that rely on testing to become flaky-test aware so as to not be misled by unreliable test failures. Beyond traditional flaky tests, this project will also (4) investigate and develop techniques to detect and repair tests that are flaky with respect to nonfunctional properties, such as execution time. Finally, given the parallels between flaky tests and difficulties in computational reproducibility, this project will also (5) develop techniques to increase the reproducibility of computational research results based on insights from the work on flaky tests.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该奖项全部或部分根据2021年美国救援计划法案（公法117-2）资助。软件对日常生活的影响越来越大，这意味着需要高质量，正确的软件。检查软件正确性的最常见方法是通过软件测试。隐含的假设是，失败的测试表明存在bug。然而，这个基本假设被重复测试的存在所否定，重复测试是在同一代码上运行时不确定地通过或失败的测试。直觉上，在被测代码或环境中有一个非确定性组件，这是预期的，甚至是期望的（因此不是软件中真正的bug），但是开发人员可能已经编写了一个不确定性测试，没有考虑这种非确定性，导致测试有时通过，有时失败。当测试套件中存在重复测试时，开发人员就不能再信任它们的测试结果。如果测试失败不再保证存在真正的bug，开发人员可能会选择忽略所有测试结果，而不是进一步调查。在这一点上，测试变得毫无用处，开发人员冒着关键错误滑入发布的软件的风险。脆弱的测试是工业界和开源社区开发人员面临的一个常见问题。该项目的目标是减轻反倾销测试的负面影响。为了实现这一目标，该项目将（1）通过利用代码进化来开发更有效地检测漏洞测试的技术，（2）开发可以减少漏洞测试失败而无需修改测试代码的专用测试环境，以及（3）增强依赖于测试的软件工程技术，以变得具有漏洞测试意识，从而不会被不可靠的测试失败所误导。除了传统的可测试性测试之外，这个项目还将（4）研究和开发技术来检测和修复在非功能属性（如执行时间）方面被测试的测试。最后，考虑到可重复性测试和计算再现性困难之间的相似性，该项目还将（5）基于可重复性测试工作的见解，开发提高计算研究结果再现性的技术。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Evolution-aware detection of order-dependent flaky tests

顺序相关片状测试的进化感知检测

• DOI: 10.1145/3533767.3534404
• 发表时间: 2022
• 期刊: 31st ACM SIGSOFT International Symposium on Software Testing and Analysis
• 作者: Li, Chengpeng;Shi, August
• 通讯作者: Shi, August