CISE Core: CCF: SHF: Small: Future-Proof Test Corpus Synthesis for Evolving Software

现代软件是复杂且不断发展的。对于代码的每一个小更改，都有引入意外后果的风险，这些后果可能会影响软件的正确性、安全性和性能。为了防止这种被称为回归错误的问题，开发人员必须在每次代码更改之后在不同的程序输入套件上测试他们的软件。然而，手工制作这样的测试输入有可能错过重要的边缘用例。这项研究正在开发自动生成测试输入的技术，以防止未来的回归。研究将集中于自动合成测试输入，这些测试输入易于维护，执行迅速，并且在检测由小代码更改引入的错误方面非常健壮。在这个项目中开发的技术旨在帮助提高关键软件系统的可靠性，减少开发期间的能源使用，并减少技术债务。此外，这项研究也有助于研究者为本科计算机科学教育开发可重复使用的课程材料，特别是通过将自动测试输入生成技术纳入课堂编程作业。项目活动本身也将为不同的本科生群体提供研究经验的机会。随机测试输入生成技术，如灰盒模糊测试，已经成功地发现了广泛使用的软件中的关键错误和安全问题。然而，传统的模糊测试需要使用数百个cpu小时生成数十亿个测试输入才能有效，这对于持续验证代码更改是不切实际的。这个项目将模糊测试研究的重点转移到生成一个可重用的回归测试输入语料库上，以支持软件进化。该研究的重点是沿着三个维度优化生成的测试输入的质量。首先，一种迭代集成模糊技术正在被开发，用于通过构造来合成简洁的测试输入。其次，突变分析被用来指导模糊测试向着综合测试输入的方向发展，这些测试输入在检测小代码更改引起的错误方面是健壮的。第三，语言建模技术被用来学习人工编写的测试输入中的常见模式。这些模型被用于开发新的模糊算法，这些算法可以合成看起来很自然的测试输入，随着软件的发展，这些输入更容易维护。这项研究的结果正以开源工具和出版物的形式传播，旨在帮助软件开发人员降低维护成本，并最终部署更可靠的软件。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Modern software is complex and continuously evolving. For every small change to the code, there is a risk of introducing unintended consequences that can affect the software's correctness, security, and performance. To guard against such issues, known as regression bugs, developers must test their software on a diverse suite of program inputs after every code change. However, manually hand-crafting such test inputs risks missing out on important corner cases. This research is developing techniques for automatically generating test inputs that guard against future regressions. The research will focus on automatically synthesizing test inputs that are easy to maintain, quick to execute, and robust at detecting faults introduced by small code changes. The technology developed in this project is intended to help improve the reliability of critical software systems, cut down energy usage during development, and reduce technical debt. Furthermore, this research is also contributing to the investigator's ongoing efforts in developing reusable course material for undergraduate computer science education, in particular, by incorporating the automatic test-input generation technology in classroom programming assignments. The project activities themselves will also provide research experience opportunities for a diverse cohort of undergraduate students.Randomized test-input generation techniques such as grey-box fuzzing have been successful at uncovering critical bugs and security issues in widely used software. However, conventional fuzz testing requires generating billions of test inputs using hundreds of CPU-hours in order to be effective, which is impractical for continuously validating code changes. This project shifts the focus of fuzz-testing research towards generating a reusable corpus of regression test inputs, in order to support software evolution. The research is focusing on optimizing the quality of the generated test inputs along three dimensions. First, an iterative ensemble fuzzing technique is being developed for synthesizing test inputs that are concise by construction. Second, mutation analysis is being used to guide fuzzing towards synthesizing test inputs that are robust at detecting faults due to small code changes. Third, language modeling techniques are being used to learn common patterns in human-authored test inputs. The models are being used to develop novel fuzzing algorithms that can synthesize natural-looking test inputs that easier to maintain as the software evolves. The results of this research are being disseminated in the form of open-source tools and publications that are intended to help software developers reduce maintenance costs and ultimately deploy more reliable software.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.