Collaborative Research: SHF: Small: Feedback-Driven Mutation Testing for Any Language

测试、确认和验证都是编程和软件工程中的核心活动。不幸的是，现有的测试技术仍然不足以在软件部署之前发现和消除关键漏洞--即使是最关键的现代软件也充满了安全漏洞和缺陷，最终导致经济每年损失数十亿美元的生产力和数据泄露。自20世纪70年代以来，一种被称为“突变测试”的技术一直在研究;它旨在帮助软件工程师同时改进他们的测试和他们的软件，通过自动向程序添加错误并检查测试套件是否可以检测到它们。虽然在理论上，这种技术对于提高软件质量是非常有效的，但有几个基本因素阻止了它在实践中的广泛应用：使用起来困难且耗时，并且现有的工具无法处理现代软件系统中部署的程序语言的多样性。该项目将解决这些挑战，并通过开发有效的工具来提高真实世界软件的质量，这些工具可以将突变测试应用于任何语言编写的程序;优先考虑工具的输出，以减少最大限度地利用它们所需的时间和精力;并将用户反馈纳入技术中，以最大限度地提高测试效率。该项目将在真实世界的开源软件（如Linux内核）上进行评估，并建立在研究人员之前的合作基础上，以大幅提高关键现实世界软件的程序和测试工作质量。该项目旨在解决的核心问题是使程序突变体在非研究环境中实用，以满足开发人员和测试工程师的需求，通过使创建或增强测试套件或同时开发代码和测试套件的人能够（1）使用“刚好足够”的突变测试来满足他们的需求，最大化在所执行的工作的交换中获得的利益，以及（2）在任何编程语言中工作，而不用担心为突变测试提供的工具支持的质量，并且不牺牲理解基于源代码的突变体的容易性，同时容易地添加针对其特定软件开发任务的自定义变异操作符。该项目的目的是适应最远点的第一个指标，以前使用的模糊错误分类的问题，最大限度地提高新奇的突变体检查的用户，以便能够快速发现未杀死的突变体，暴露严重的缺陷，在测试或验证工作。然而，仅仅是新奇是不够的：反馈驱动的突变测试还必须帮助用户避免无关紧要的、等同的突变体，杀死处于优势层次的突变体，并且（最重要的是）结合用户反馈。 如果用户将一个突变体标记为无关紧要的，或等同的，或（特别是）高影响，那么该信息也必须用于通知未来突变体的排名。为了使这种方法最大限度地发挥价值，该项目还建议改进源代码级多语言突变体生成的最新技术，使用户能够轻松地为新的编程语言生成突变体，该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的评估，被认为值得支持。影响审查标准。

Testing, validation, and verification are all central activities in programming and software engineering. Unfortunately, existing techniques for testing remain inadequate for finding and eliminating key vulnerabilities before software deployment -- even the most critical modern software is rife with security vulnerabilities and defects that ultimately cost the economy billions of dollars annually in lost productivity and compromised data. A technique known as "mutation testing" has been researched since the 1970s; it aims to help software engineers improve their tests and their software at the same time, by automatically adding bugs to a program and checking whether the test suite can detect them. Although in theory this technique is extremely effective for improving software quality, there are several fundamental factors that prevent it from being widely used in practice: it is difficult and time-consuming to use, and the tools that exist for it cannot all handle the diversity of program languages that are deployed in modern software systems. This project will tackle these challenges and allow this important technique to be used to improve quality of real-world software by developing efficient tools that can apply mutation testing to programs written in any language; prioritize the output of the tools to reduce the amount of time and effort needed to make maximal use of them; and incorporate user feedback into the technique to maximize testing efficiency. The project will be evaluated on real-world open source software like the Linux kernel, and build on the researchers' previous collaborations to substantially improve program and test effort quality on critical real-world software.The core problem this project aims to address is making program mutants practical in nonresearch settings, in a way that meets the needs of developers and test engineers, by making it possible for someone creating or enhancing a test suite, or developing code and test suite in tandem, to (1) use "just enough" mutation testing for their needs, maximizing benefit gained in exchange for work performed, and (2) to work in any programming language without worrying about the quality of tool support provided for mutation testing, and without sacrificing the ease of understanding of source-based mutants, while easily adding custom mutation operators that target their specific software development task. This project aims to adapt the Furthest-Point-First metric previously used in fuzzer bug triaging to the problem of maximizing the novelty of mutants examined by a user, in order to make it possible to quickly discover unkilled mutants that expose serious defects in a testing or verification effort. However, novelty alone is not sufficient: feedback-driven mutation testing must also help users avoid inconsequential, equivalent mutants, kill mutants high in the dominance hierarchy, and (most importantly) incorporate user feedback. If a user marks a mutant as inconsequential, or equivalent, or (especially) high impact, then that information must be used to inform the ranking of future mutants as well. In order to make such an approach maximally valuable, this project also proposes to improve the state-of-the-art in source-level multilingual mutant generation, allowing users to easily generate mutants for new programming languages, or even for custom DSLs that are part of a specific project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.