CAREER: Enhanced Reliability and Efficiency of Software Regression Testing in the Presence of Flaky Tests

软件的开发通常是在一个持续的开发和集成过程中进行的，该过程包括导致软件连续发布的增量更改，每个发布都经过严格的软件测试，以检查最近的代码更改是否破坏了现有功能。这个过程被称为回归测试，广泛用于软件开发实践。测试用例生成中的一个主要问题是重复测试的出现：在同一版本的代码上不确定地通过或失败的测试。测试失败可能会误导开发人员最近的更改，浪费开发人员的时间，并降低开发人员对软件测试的信任。许多软件开发组织都报告说，可靠性测试是他们最大的问题之一，因为它们混淆了保证目标。 该项目旨在提高回归测试的可靠性和效率，在现有的可重复性测试。它将产生旨在高效和有效地解决固有的非决定性的工具。 工作重点是（1）通过预测重要的测试属性来降低脆弱性测试检测和调试技术的成本，（2）开发新技术来预测脆弱性相关属性，（3）加速和减少回归测试所需的资源，（4）开发新技术来系统地检测脆弱性测试，以及（5）减少Android用户界面测试中的脆弱性。该项目还将编制关于面对不确定性的方案编制专题的教育和培训课程，并将与工业界合作转让技术。对片状的研究将从典型的黑盒方法转向新的水平，通过新颖的白盒和基于学习的方法进行检测，调试和修复。该方法将使用静态和动态分析来计算状态污染，这可能会影响基于测试执行顺序的测试片状。该工作涉及组合设计理论，以提高效率的顺序相关的测试检测。将特别注意图形用户界面中使用记录-重放和测试输入生成的重复测试。 测试覆盖率计算可用于预测代码更改是否会影响测试的输出，将使用机器学习方法。 该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估来支持。

Software is usually developed in a continuous development and integration process that incorporates incremental changes leading to successive releases of the software, where each release undergoes rigorous software testing to check whether recent code changes had broken existing functionalities. This process, known as regression testing, is widely used in software development practice. A major problem in the generation of test cases is the presence of flaky tests: tests that non-deterministically pass or fail on the same version of the code. Failures from flaky tests can mislead developers about their recent changes, waste developers’ time, and reduce developers’ trust in software testing. Many software development organizations have reported that flaky tests are one of their biggest problems, because they confound assurance goals. This project aims to improve the reliability and efficiency of regression testing in the presence of flaky tests. It will produce tools that aim to be efficient and effective at resolving the inherent nondeterminism. The work focuses on (1) reducing the cost of flaky-test detection and debugging techniques by predicting important test properties, (2) developing new techniques to predict flakiness-related properties, (3) speeding up and reducing the resources needed by regression testing, (4) developing new techniques to systematically detect flaky tests, and (5) reducing the flakiness in Android user interface testing. The project will also produce curriculum for education and training on the topic of programming in the face of nondeterminism, and will work with industry to transfer technology. The research on flakiness will move from the typical, black-box approaches to a new level for detecting, debugging, and fixing through novel, white-box and learning-based approaches. The approach will use static and dynamic analyses to compute state pollution, which may affect test flakiness based on the test execution order. The work involves combinatorial design theory to improve the efficiency of order-dependent test detection. Special attention will be paid to flaky tests in graphical user interfaces using record-and-replay and test input generation. Test coverage computations, which can be used to predict whether a code change will affect the test's output, will use a machine learning approach. The work will result in tool implementations and large-scale evaluations in open source and proprietary environments.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.