Machine Learning for Efficient Regression Testing in Continuous Integration Context

在持续集成环境中进行高效回归测试的机器学习

• 批准号: RGPIN-2022-05131
• 负责人: Kahani, Nafiseh
• 金额: $ 1.82万
• 依托单位: Carleton University
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=765861
• 关键词: Machine; Learning; Efficient; Regression; Testing

Continuous Integration (CI) has become one of the main enablers of automated and modern software engineering and has been widely adopted by IT organizations and open-source communities. CI systems allow software developers to submit their code into a shared repository on a frequent basis. Each submission is then built, analyzed, and tested in an automated process. CI provides early feedback to developers, reduces integration risks, and facilitates the early detection of faults. Despite the benefits, CI systems impose a high operational and computational cost that requires significant investment to operate, which threatens the benefits of CI. For example, it has been reported that running CI systems in Google and Microsoft costs these companies millions of dollars. Automated regression testing is crucial to ensure the quality of software systems in CI. However, it is also one of the main reasons for its high costs. For large software projects, software developers need to spend a significant amount of time developing and maintaining many regression test cases (maintainability issues), the execution of which requires significant computational resources (execution issues).   The proposed research program will use state-of-the-art Machine Learning (ML) and software analysis techniques to develop automated, practical, and accurate solutions to improve regression testing efficiency in the context of CI while considering maintainability and execution issues. Test Case Prioritization (TCP) techniques deal with the costly execution of regression testing by prioritizing the specific test cases so that faults may be detected as early as possible. Many TSP techniques have been proposed in the context of regression testing using ML techniques. However, even the best of them cannot achieve satisfactory results for most software systems, arguably due to imbalanced and noisy training datasets. Thus, we will first explore different techniques to account for imbalanced and noisy datasets in order to improve the accuracy of ML-based TCP. We will then explore static analysis, natural language processing, and ML techniques to automatically detect and repair obsolete test cases that are not updated to account for new changes in the software being tested. As, obsolete test cases require significant maintenance efforts from the development team due to the ever-changing codebase in the context of CI. Due to the wide adoption of CI, we believe the result of this research program will help multiple Canadian and international companies decrease the computational and operational costs of their CI systems. This research program will also enable Highly Qualified Personnel (HQP) to acquire practical knowledge on ML, software testing and analysis, and continuous integration. IT organizations increasingly search for HQP who can understand software engineering and ML in order to improve the software development process, ensuring the desirability of HQP from this program.

持续集成（CI）已经成为自动化和现代软件工程的主要推动者之一，并已被IT组织和开源社区广泛采用。CI系统允许软件开发人员频繁地将他们的代码提交到共享存储库中。然后，在自动化过程中构建、分析和测试每个提交。CI为开发人员提供早期反馈，降低集成风险，并促进故障的早期检测。 尽管有这些好处，CI系统强加了高的操作和计算成本，需要大量的投资来操作，这威胁到CI的好处。例如，据报道，在谷歌和微软运行CI系统花费了这些公司数百万美元。自动化回归测试对于保证CI软件系统的质量至关重要。这也是其成本高的主要原因之一。对于大型软件项目，软件开发人员需要花费大量的时间开发和维护许多回归测试用例（可维护性问题），这些测试用例的执行需要大量的计算资源（执行问题）。 拟议的研究计划将使用最先进的机器学习（ML）和软件分析技术来开发自动化，实用和准确的解决方案，以提高CI背景下的回归测试效率，同时考虑可维护性和执行问题。测试用例优先级划分（TCP）技术通过对特定测试用例进行优先级划分来处理回归测试的高成本执行，以便尽可能早地检测到故障。在使用ML技术进行回归测试的背景下，已经提出了许多TSP技术。然而，对于大多数软件系统来说，即使是其中最好的也不能达到令人满意的结果，这可能是由于训练数据集的不平衡和噪声。因此，我们将首先探索不同的技术来解决不平衡和噪声数据集，以提高基于ML的TCP的准确性。然后，我们将探索静态分析，自然语言处理和ML技术，以自动检测和修复未更新的过时测试用例，以应对正在测试的软件中的新更改。因为，过时的测试用例需要开发团队进行大量的维护工作，因为CI环境中的代码库不断变化。由于CI的广泛采用，我们相信这项研究计划的结果将有助于多家加拿大和国际公司降低其CI系统的计算和运营成本。该研究计划还将使高素质人员（HQP）能够获得ML、软件测试和分析以及持续集成方面的实用知识。IT组织越来越多地寻找能够理解软件工程和ML的HQP，以改进软件开发过程，确保HQP从该计划中的可取性。