SHF: Medium: Collaborative Research: Enhancing Continuous Integration Testing for the Open-Source Ecosystem

SHF：媒介：协作研究：加强开源生态系统的持续集成测试

• 批准号: 2100037
• 负责人: Jonathan Bell
• 金额: $ 32.84万
• 依托单位: Northeastern University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-09-01 至 2023-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2100037&HistoricalAwards=false
• 关键词: SHF; Medium; Collaborative; Research; Enhancing

Continuous integration (CI) is an important software development activity that aims to improve software development by automating software compilation and regression testing. Recent studies report that CI helps developers deploy faster and reduce development cost. Given these success stories, CI has attracted rapidly increasing interest and adoption, e.g., Travis CI, the currently most popular CI service, is used by over 300,000 GitHub projects. Despite the success of CI, developers report they would like to see improvements in CI. First, they want to faster obtain regression test results. Second, they want better handling of so-called flaky tests, which are regression tests that can non-deterministically pass or fail, and whose failures negatively affect developer's productivity. Third, developers report that CI builds do not provide sufficient debugging assistance for reasoning about failed regression tests. While regression testing has been studied for over three decades, it has not been studied in the context of CI until recently.To substantially improve regression testing in CI, the PIs propose to develop novel techniques and tools that address three important challenges: (1) test selection to speed up regression testing and the development cycle, (2) test reliability to mitigate the problems that flaky tests introduce, and (3) debugging assistance to ease the effort of diagnosing and fixing the true and flaky regression test failures. The PIs plan to develop techniques and tools based on a mix of static and dynamic program analyses, leveraging not only information from two project revisions (as traditional in regression testing) but also from all historical build and testing information available in CI testing. The PIs plan to embody their techniques in a tool-set and evaluate them extensively on open-source projects and in industrial collaborations. The broader impacts of enhancing continuous integration testing are to allow software developers to faster build higher quality software, which can benefit our modern society that greatly depends on software.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

持续集成（CI）是一种重要的软件开发活动，旨在通过自动化软件编译和回归测试来改进软件开发。 最近的研究表明，CI可以帮助开发人员更快地部署并降低开发成本。鉴于这些成功的故事，CI已经吸引了迅速增长的兴趣和采用，例如，Travis CI是目前最受欢迎的CI服务，被超过30万个GitHub项目使用。 尽管CI取得了成功，但开发人员报告说，他们希望看到CI的改进。首先，他们希望更快地获得回归测试结果。 其次，他们希望更好地处理所谓的回归测试，回归测试可以不确定地通过或失败，其失败会对开发人员的生产力产生负面影响。 第三，开发人员报告说，CI构建没有为失败的回归测试提供足够的调试帮助。 虽然回归测试已经研究了三十多年，但直到最近才在CI的背景下进行研究。为了大大改善CI中的回归测试，PI建议开发新的技术和工具，以解决三个重要挑战：（1）测试选择，以加快回归测试和开发周期，（2）测试可靠性，以减轻重复测试引入的问题，以及（3）调试帮助，以减轻诊断和修复真实和非真实回归测试失败的工作。PI计划开发基于静态和动态程序分析的技术和工具，不仅利用来自两个项目修订版的信息（如回归测试中的传统信息），还利用CI测试中可用的所有历史构建和测试信息。PI计划将他们的技术体现在一个工具集中，并在开源项目和工业合作中对其进行广泛评估。增强持续集成测试的更广泛影响是允许软件开发人员更快地构建更高质量的软件，这可以使我们极大地依赖软件的现代社会受益。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

A retrospective study of one decade of artifact evaluations

十年工件评估的回顾性研究

• DOI: 10.1145/3540250.3549172
• 发表时间: 2022
• 期刊: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
• 作者: Winter, Stefan;Timperley, Christopher S.;Hermann, Ben;Cito, Jürgen;Bell, Jonathan;Hilton, Michael;Beyer, Dirk
• 通讯作者: Beyer, Dirk

A Tool for Mutation Analysis in Racket

Racket 中的突变分析工具

• DOI: 10.1109/icstw58534.2023.00061
• 发表时间: 2023
• 期刊: IEEE
• 作者: Zhuang, Bambi;Perretta, James;Guha, Arjun;Bell, Jonathan
• 通讯作者: Bell, Jonathan

Flaky Test Dataset to Accompany "FlakeFlagger: Predicting Flakiness Without Rerunning Tests"

伴随“FlakeFlagger：无需重新运行测试即可预测不稳定”的不稳定测试数据集

• DOI: 10.5281/zenodo.4450722
• 发表时间: 2021
• 期刊: Zenodo
• 作者: Alshammari, Abdulrahman;Morris, Christopher;Hilton, Michael;Bell, Jonathan
• 通讯作者: Bell, Jonathan

CONFETTI: Amplifying Concolic Guidance for Fuzzers

• DOI: 10.1145/3510003.3510628
• 发表时间: 2022-05
• 期刊: 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE)
• 作者: J. Kukucka;Luís Pina;P. Ammann;Jonathan Bell

The Effects of Computational Resources on Flaky Tests (Artifact)

计算资源对 Flaky 测试的影响（工件）

• DOI: 10.5281/zenodo.10015434
• 发表时间: 2023
• 期刊: Zenodo
• 作者: Silva, Denini;Gruber, Martin;Gokhale, Satyajit;Arteca, Ellen;Turcotte, Alexi;D'Amorim, Marcelo;Lam, Wing;Winter, Stefan;Bell, Jonathan
• 通讯作者: Bell, Jonathan