CISE Core: CCF: SHF: Small: Future-Proof Test Corpus Synthesis for Evolving Software

现代软件是复杂的，并不断发展。对于代码的每一个小的更改，都有可能引入意想不到的后果，从而影响软件的正确性，安全性和性能。为了防止这些问题，即所谓的回归错误，开发人员必须在每次代码更改后在不同的程序输入套件上测试他们的软件。然而，手动手工制作这样的测试输入有可能错过重要的角落案例。这项研究正在开发自动生成测试输入的技术，以防止未来的回归。该研究将集中于自动合成测试输入，这些输入易于维护，快速执行，并且在检测由小代码更改引入的故障时具有鲁棒性。该项目开发的技术旨在帮助提高关键软件系统的可靠性，减少开发过程中的能源使用，并减少技术债务。此外，这项研究也有助于调查员的持续努力，在开发可重复使用的课程材料本科计算机科学教育，特别是通过将自动测试输入生成技术在课堂编程作业。项目活动本身也将为不同的本科生群体提供研究经验的机会。随机测试输入生成技术，如灰盒模糊测试，已经成功地发现了广泛使用的软件中的关键错误和安全问题。然而，传统的模糊测试需要使用数百个CPU小时来生成数十亿个测试输入才能有效，这对于连续验证代码更改是不切实际的。该项目将模糊测试研究的重点转向生成可重用的回归测试输入语料库，以支持软件演化。研究的重点是优化质量的生成测试输入沿沿着三个维度。首先，一个迭代集成模糊技术正在开发合成测试输入，是简洁的建设。第二，突变分析被用来指导模糊合成测试输入，是强大的检测故障，由于小的代码变化。第三，语言建模技术被用来学习人类编写的测试输入中的常见模式。这些模型被用于开发新的模糊算法，这些算法可以合成看起来很自然的测试输入，随着软件的发展，这些输入更容易维护。 这项研究的成果正在以开源工具和出版物的形式传播，旨在帮助软件开发人员降低维护成本，并最终部署更可靠的软件。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Modern software is complex and continuously evolving. For every small change to the code, there is a risk of introducing unintended consequences that can affect the software's correctness, security, and performance. To guard against such issues, known as regression bugs, developers must test their software on a diverse suite of program inputs after every code change. However, manually hand-crafting such test inputs risks missing out on important corner cases. This research is developing techniques for automatically generating test inputs that guard against future regressions. The research will focus on automatically synthesizing test inputs that are easy to maintain, quick to execute, and robust at detecting faults introduced by small code changes. The technology developed in this project is intended to help improve the reliability of critical software systems, cut down energy usage during development, and reduce technical debt. Furthermore, this research is also contributing to the investigator's ongoing efforts in developing reusable course material for undergraduate computer science education, in particular, by incorporating the automatic test-input generation technology in classroom programming assignments. The project activities themselves will also provide research experience opportunities for a diverse cohort of undergraduate students.Randomized test-input generation techniques such as grey-box fuzzing have been successful at uncovering critical bugs and security issues in widely used software. However, conventional fuzz testing requires generating billions of test inputs using hundreds of CPU-hours in order to be effective, which is impractical for continuously validating code changes. This project shifts the focus of fuzz-testing research towards generating a reusable corpus of regression test inputs, in order to support software evolution. The research is focusing on optimizing the quality of the generated test inputs along three dimensions. First, an iterative ensemble fuzzing technique is being developed for synthesizing test inputs that are concise by construction. Second, mutation analysis is being used to guide fuzzing towards synthesizing test inputs that are robust at detecting faults due to small code changes. Third, language modeling techniques are being used to learn common patterns in human-authored test inputs. The models are being used to develop novel fuzzing algorithms that can synthesize natural-looking test inputs that easier to maintain as the software evolves. The results of this research are being disseminated in the form of open-source tools and publications that are intended to help software developers reduce maintenance costs and ultimately deploy more reliable software.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.