CISE Core: CCF: SHF: Small: Future-Proof Test Corpus Synthesis for Evolving Software

CISE 核心：CCF：SHF：小型：面向发展软件的面向未来的测试语料库合成

• 批准号: 2120955
• 负责人: Rohan Padhye
• 金额: $ 54.61万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2120955&HistoricalAwards=false
• 关键词: CISE; Core; CCF; SHF; Small

Modern software is complex and continuously evolving. For every small change to the code, there is a risk of introducing unintended consequences that can affect the software's correctness, security, and performance. To guard against such issues, known as regression bugs, developers must test their software on a diverse suite of program inputs after every code change. However, manually hand-crafting such test inputs risks missing out on important corner cases. This research is developing techniques for automatically generating test inputs that guard against future regressions. The research will focus on automatically synthesizing test inputs that are easy to maintain, quick to execute, and robust at detecting faults introduced by small code changes. The technology developed in this project is intended to help improve the reliability of critical software systems, cut down energy usage during development, and reduce technical debt. Furthermore, this research is also contributing to the investigator's ongoing efforts in developing reusable course material for undergraduate computer science education, in particular, by incorporating the automatic test-input generation technology in classroom programming assignments. The project activities themselves will also provide research experience opportunities for a diverse cohort of undergraduate students.Randomized test-input generation techniques such as grey-box fuzzing have been successful at uncovering critical bugs and security issues in widely used software. However, conventional fuzz testing requires generating billions of test inputs using hundreds of CPU-hours in order to be effective, which is impractical for continuously validating code changes. This project shifts the focus of fuzz-testing research towards generating a reusable corpus of regression test inputs, in order to support software evolution. The research is focusing on optimizing the quality of the generated test inputs along three dimensions. First, an iterative ensemble fuzzing technique is being developed for synthesizing test inputs that are concise by construction. Second, mutation analysis is being used to guide fuzzing towards synthesizing test inputs that are robust at detecting faults due to small code changes. Third, language modeling techniques are being used to learn common patterns in human-authored test inputs. The models are being used to develop novel fuzzing algorithms that can synthesize natural-looking test inputs that easier to maintain as the software evolves. The results of this research are being disseminated in the form of open-source tools and publications that are intended to help software developers reduce maintenance costs and ultimately deploy more reliable software.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

现代软件是复杂的，而且还在不断发展。对于代码的每一处微小更改，都存在引入意外后果的风险，这些后果可能会影响软件的正确性、安全性和性能。为了防止这种被称为回归错误的问题，开发人员必须在每次代码更改后，在不同的程序输入套件上测试他们的软件。然而，手工制作这样的测试输入可能会错过重要的转角案例。这项研究正在开发自动生成测试输入的技术，以防止未来的回归。这项研究将专注于自动合成测试输入，这些输入易于维护、快速执行，并且在检测由微小代码更改引入的故障时具有健壮性。该项目开发的技术旨在帮助提高关键软件系统的可靠性，减少开发过程中的能源消耗，并减少技术债务。此外，这项研究还有助于研究人员为本科计算机科学教育开发可重复使用的课程材料，特别是通过将自动测试输入生成技术纳入课堂编程作业。项目活动本身也将为不同的本科生提供研究经验的机会。随机测试输入生成技术，如灰箱模糊，已经成功地发现了广泛使用的软件中的关键错误和安全问题。然而，传统的模糊测试需要使用数百个CPU小时生成数十亿个测试输入才能有效，这对于持续验证代码更改是不切实际的。该项目将模糊测试研究的重点转移到生成可重用的回归测试输入语料库，以支持软件演化。这项研究的重点是沿三个维度优化生成的测试输入的质量。首先，正在开发一种迭代集成模糊技术，用于合成通过构造而简洁的测试输入。其次，突变分析正被用来指导模糊合成测试输入，这些输入在检测由于微小的代码变化而产生的错误时是健壮的。第三，语言建模技术被用来学习人类编写的测试输入中的常见模式。这些模型正被用来开发新的模糊算法，这些算法可以合成看起来自然的测试输入，随着软件的发展更容易维护。这项研究的结果以开源工具和出版物的形式传播，旨在帮助软件开发人员降低维护成本并最终部署更可靠的软件。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

On the Naturalness of Fuzzer-Generated Code

关于模糊器生成代码的自然性

• DOI: 10.1145/3524842.3527972
• 发表时间: 2022
• 期刊: 19th International Conference on Mining Software Repositories
• 作者: Kambhamettu, Rajeswari Hita;Billos, John;Oluwaseun-Apo, Tomi;Gafford, Benjamin;Padhye, Rohan;Hellendoorn, Vincent J.
• 通讯作者: Hellendoorn, Vincent J.

Guiding Greybox Fuzzing with Mutation Testing

• DOI: 10.1145/3597926.3598107
• 发表时间: 2023-07
• 期刊: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis
• 作者: Vasudev Vikram;Isabella Laybourn;Ao Li;Nicole Nair;Kelton OBrien;Rafaello Sanna;Rohan Padhye