SHF: Medium: Collaborative Research: Enhancing Continuous Integration Testing for the Open-Source Ecosystem

持续集成(CI)是一项重要的软件开发活动，旨在通过自动化软件编译和回归测试来改进软件开发。最近的研究报告称，CI帮助开发人员更快地部署并降低开发成本。鉴于这些成功的案例，CI吸引了越来越多的人的兴趣和采用，例如，目前最受欢迎的CI服务Travis CI被30多万个GitHub项目使用。尽管CI取得了成功，但开发人员报告说，他们希望看到CI的改进。首先，他们希望更快地获得回归测试结果。其次，他们希望更好地处理所谓的薄片测试，这是一种回归测试，可以不确定地通过或失败，其失败会对开发人员的生产力产生负面影响。第三，开发人员报告说，CI构建不能为失败的回归测试提供足够的调试帮助。回归测试已经被研究了三十多年，但直到最近才在CI的背景下进行研究。为了显著改善CI中的回归测试，PI建议开发新的技术和工具来解决三个重要挑战：(1)测试选择以加快回归测试和开发周期；(2)测试可靠性以缓解片状测试带来的问题；(3)调试辅助以简化诊断和修复真实和片状回归测试失败的工作。PI计划开发基于静态和动态程序分析的技术和工具，不仅利用来自两个项目修订版的信息(回归测试中的传统)，而且利用CI测试中可用的所有历史构建和测试信息。私人投资机构计划将他们的技术体现在一个工具集中，并在开源项目和工业合作中对其进行广泛的评估。加强持续集成测试的更广泛影响是允许软件开发人员更快地构建更高质量的软件，这可以使我们严重依赖软件的现代社会受益。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Continuous integration (CI) is an important software development activity that aims to improve software development by automating software compilation and regression testing. Recent studies report that CI helps developers deploy faster and reduce development cost. Given these success stories, CI has attracted rapidly increasing interest and adoption, e.g., Travis CI, the currently most popular CI service, is used by over 300,000 GitHub projects. Despite the success of CI, developers report they would like to see improvements in CI. First, they want to faster obtain regression test results. Second, they want better handling of so-called flaky tests, which are regression tests that can non-deterministically pass or fail, and whose failures negatively affect developer's productivity. Third, developers report that CI builds do not provide sufficient debugging assistance for reasoning about failed regression tests. While regression testing has been studied for over three decades, it has not been studied in the context of CI until recently.To substantially improve regression testing in CI, the PIs propose to develop novel techniques and tools that address three important challenges: (1) test selection to speed up regression testing and the development cycle, (2) test reliability to mitigate the problems that flaky tests introduce, and (3) debugging assistance to ease the effort of diagnosing and fixing the true and flaky regression test failures. The PIs plan to develop techniques and tools based on a mix of static and dynamic program analyses, leveraging not only information from two project revisions (as traditional in regression testing) but also from all historical build and testing information available in CI testing. The PIs plan to embody their techniques in a tool-set and evaluate them extensively on open-source projects and in industrial collaborations. The broader impacts of enhancing continuous integration testing are to allow software developers to faster build higher quality software, which can benefit our modern society that greatly depends on software.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.