Collaborative Research: SHF: Small: Feedback-Driven Mutation Testing for Any Language

测试、确认和验证都是编程和软件工程中的中心活动。不幸的是，现有的测试技术仍然不足以在软件部署之前发现和消除关键漏洞--即使是最关键的现代软件也充斥着安全漏洞和缺陷，最终每年会给经济造成数十亿美元的生产力损失和数据泄露。自20世纪70年代以来，人们一直在研究一种名为“突变测试”的技术；它旨在帮助软件工程师同时改进他们的测试和软件，方法是自动将错误添加到程序中，并检查测试套件是否可以检测到这些错误。尽管从理论上讲，这种技术对提高软件质量非常有效，但有几个基本因素阻碍了它在实践中的广泛应用：使用起来既困难又耗时，而且现有的工具不能完全处理现代软件系统中部署的编程语言的多样性。该项目将解决这些挑战，并允许使用这一重要技术来提高现实世界软件的质量，方法是开发有效的工具，将突变测试应用于以任何语言编写的程序；对工具的输出进行优先排序，以减少最大限度地使用工具所需的时间和工作量；并将用户反馈纳入该技术，以最大限度地提高测试效率。该项目将在Linux内核等现实世界的开源软件上进行评估，并建立在研究人员之前的合作基础上，以显著提高关键现实软件上的程序和测试工作质量。该项目旨在解决的核心问题是使程序突变在非研究环境中具有实用性，以满足开发人员和测试工程师的需求，通过使某人能够创建或增强测试套件，或同时开发代码和测试套件，(1)对其需求使用“恰到好处”的突变测试，最大限度地利用所执行的工作所获得的好处，以及(2)在任何编程语言中工作，而不担心为突变测试提供的工具支持的质量，并且不牺牲基于源代码的突变体的易解性，同时容易地添加针对其特定软件开发任务的定制突变操作符。该项目旨在将以前在Fuzzer错误分类中使用的最远点优先度量应用于最大化用户检查的突变体的新颖性的问题，以便能够快速发现在测试或验证工作中暴露严重缺陷的未被杀死的突变体。然而，仅有新颖性是不够的：反馈驱动的突变测试还必须帮助用户避免无关紧要的、等价的突变，杀死处于主导地位的突变，以及(最重要的)纳入用户反馈。如果用户将一个突变体标记为无关紧要、同等或(特别是)高影响，则该信息也必须用于通知未来突变体的排名。为了使这种方法具有最大的价值，该项目还建议改进源码级别多语言突变生成的最新技术，允许用户轻松地为新的编程语言生成突变，甚至为特定项目中的定制DSL生成突变。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Testing, validation, and verification are all central activities in programming and software engineering. Unfortunately, existing techniques for testing remain inadequate for finding and eliminating key vulnerabilities before software deployment -- even the most critical modern software is rife with security vulnerabilities and defects that ultimately cost the economy billions of dollars annually in lost productivity and compromised data. A technique known as "mutation testing" has been researched since the 1970s; it aims to help software engineers improve their tests and their software at the same time, by automatically adding bugs to a program and checking whether the test suite can detect them. Although in theory this technique is extremely effective for improving software quality, there are several fundamental factors that prevent it from being widely used in practice: it is difficult and time-consuming to use, and the tools that exist for it cannot all handle the diversity of program languages that are deployed in modern software systems. This project will tackle these challenges and allow this important technique to be used to improve quality of real-world software by developing efficient tools that can apply mutation testing to programs written in any language; prioritize the output of the tools to reduce the amount of time and effort needed to make maximal use of them; and incorporate user feedback into the technique to maximize testing efficiency. The project will be evaluated on real-world open source software like the Linux kernel, and build on the researchers' previous collaborations to substantially improve program and test effort quality on critical real-world software.The core problem this project aims to address is making program mutants practical in nonresearch settings, in a way that meets the needs of developers and test engineers, by making it possible for someone creating or enhancing a test suite, or developing code and test suite in tandem, to (1) use "just enough" mutation testing for their needs, maximizing benefit gained in exchange for work performed, and (2) to work in any programming language without worrying about the quality of tool support provided for mutation testing, and without sacrificing the ease of understanding of source-based mutants, while easily adding custom mutation operators that target their specific software development task. This project aims to adapt the Furthest-Point-First metric previously used in fuzzer bug triaging to the problem of maximizing the novelty of mutants examined by a user, in order to make it possible to quickly discover unkilled mutants that expose serious defects in a testing or verification effort. However, novelty alone is not sufficient: feedback-driven mutation testing must also help users avoid inconsequential, equivalent mutants, kill mutants high in the dominance hierarchy, and (most importantly) incorporate user feedback. If a user marks a mutant as inconsequential, or equivalent, or (especially) high impact, then that information must be used to inform the ranking of future mutants as well. In order to make such an approach maximally valuable, this project also proposes to improve the state-of-the-art in source-level multilingual mutant generation, allowing users to easily generate mutants for new programming languages, or even for custom DSLs that are part of a specific project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.