CAREER: Enhanced Reliability and Efficiency of Software Regression Testing in the Presence of Flaky Tests

软件通常是在持续的开发和集成过程中开发的，该过程结合了导致软件连续发布的增量更改，其中每个发布都要经过严格的软件测试，以检查最近的代码更改是否破坏了现有功能。这个过程被称为回归测试，在软件开发实践中被广泛使用。测试用例生成中的一个主要问题是存在不可靠的测试：在相同版本的代码上不确定通过或失败的测试。不可靠的测试失败可能会误导开发人员对他们最近的更改，浪费开发人员的时间，并降低开发人员对软件测试的信任。许多软件开发组织报告说，不可靠的测试是他们最大的问题之一，因为它们混淆了保证目标。这个项目的目的是在存在片状测试的情况下提高回归测试的可靠性和效率。它将产生旨在高效和有效地解决固有的不确定性的工具。这项工作的重点是(1)通过预测重要的测试特性来降低片状测试检测和调试技术的成本，(2)开发预测与片状相关的特性的新技术，(3)加快和减少回归测试所需的资源，(4)开发新的技术来系统地检测片状测试，以及(5)减少Android用户界面测试中的片状。该项目还将编制关于面对不确定性的编程主题的教育和培训课程，并将与业界合作转让技术。对剥落的研究将通过新颖、白盒和基于学习的方法，从典型的黑盒方法转移到一个新的检测、调试和修复的水平。该方法将使用静态和动态分析来计算状态污染，根据测试执行顺序，状态污染可能会影响测试的片状。该工作涉及到组合设计理论，以提高顺序相关测试检测的效率。将特别注意使用记录和回放以及测试输入生成的图形用户界面中的不可靠测试。可用于预测代码更改是否会影响测试输出的测试覆盖率计算将使用机器学习方法。这项工作将导致在开源和专有环境中实施工具和进行大规模评估。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Software is usually developed in a continuous development and integration process that incorporates incremental changes leading to successive releases of the software, where each release undergoes rigorous software testing to check whether recent code changes had broken existing functionalities. This process, known as regression testing, is widely used in software development practice. A major problem in the generation of test cases is the presence of flaky tests: tests that non-deterministically pass or fail on the same version of the code. Failures from flaky tests can mislead developers about their recent changes, waste developers’ time, and reduce developers’ trust in software testing. Many software development organizations have reported that flaky tests are one of their biggest problems, because they confound assurance goals. This project aims to improve the reliability and efficiency of regression testing in the presence of flaky tests. It will produce tools that aim to be efficient and effective at resolving the inherent nondeterminism. The work focuses on (1) reducing the cost of flaky-test detection and debugging techniques by predicting important test properties, (2) developing new techniques to predict flakiness-related properties, (3) speeding up and reducing the resources needed by regression testing, (4) developing new techniques to systematically detect flaky tests, and (5) reducing the flakiness in Android user interface testing. The project will also produce curriculum for education and training on the topic of programming in the face of nondeterminism, and will work with industry to transfer technology. The research on flakiness will move from the typical, black-box approaches to a new level for detecting, debugging, and fixing through novel, white-box and learning-based approaches. The approach will use static and dynamic analyses to compute state pollution, which may affect test flakiness based on the test execution order. The work involves combinatorial design theory to improve the efficiency of order-dependent test detection. Special attention will be paid to flaky tests in graphical user interfaces using record-and-replay and test input generation. Test coverage computations, which can be used to predict whether a code change will affect the test's output, will use a machine learning approach. The work will result in tool implementations and large-scale evaluations in open source and proprietary environments.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.