Collaborative Research: SHF: Small: Automated Quantitative Assessment of Testing Difficulty

我们的社会严重依赖于在我们周围越来越多的可编程设备上运行的软件系统。此外，汽车和飞机等安全关键系统中的软件数量不断增加。软件质量保证是我们在这个现代计算主导的时代面临的最基本的问题之一。人们每天都可以在新闻中读到由质量差的软件引起的可靠性和安全问题。开发能够在软件系统运行期间造成灾难性后果之前提高软件系统质量的技术是极其重要的。最常见的软件质量保证技术是软件测试。尽管自动化软件测试技术取得了巨大的进步，但很难预测它们的有效性。给定一个软件，没有现有的技术可以预测自动测试该软件的挑战性。在这个项目中，目标是开发评估自动测试软件的难度的技术。现有的软件复杂性度量不能提供有意义的测试难度评估。该项目的目标是开发可扩展的技术，可以提供测试难度的定量评估。为了具有可扩展性和实用性，该方法必须依赖于提供有效分析的抽象级别，同时保留与测试难度相关的程序行为的有意义特征。这个项目中使用的方法建立在两个概念上，这两个概念为测试难度的定量评估提供了一个有前途的抽象：1）路径复杂性，2）路径选择性。路径复杂性评估给定程序中的路径数量如何随着执行深度的增加而增加，路径选择性评估找到满足路径条件的值的难度。从事该项目的研究人员团队将开发自动计算路径复杂性和路径选择性的技术，然后将它们联合收割机结合起来，以获得测试难度的定量测量。通过开发能够评估软件测试难度的技术，该项目将能够在更好地分配资源用于软件质量保证任务的基础上开发更有效的软件测试技术。这将导致软件质量的提高，并减少导致可靠性和安全问题的软件缺陷。其次，研究活动将有助于使研究生和本科生接触软件质量保证的挑战和技术。最后，研究活动将有助于传播知识，技术和工具的范围内开发的项目，通过出版在公开文献和提供软件工具，是作为开放源代码开发的。这个奖项反映了NSF的法定使命，并已被认为是值得的支持，通过评估使用基金会的智力价值和更广泛的影响审查标准。

Our society is heavily reliant on software systems running on an increasingly large number of programmable devices that surround us. Moreover, the amount of software in safety-critical systems such as cars and planes keeps increasing. Software-quality assurance is one of the most fundamental problems that we are facing in this modern computing-dominated era. One can read about dependability and security problems caused by poor-quality software in the news everyday. It is extremely crucial to develop techniques that can improve the quality of software systems before they cause disastrous consequences during operation. The most common software-quality assurance technique is software testing. Although there has been a surge of progress in automated software-testing techniques, it is hard to predict their effectiveness. Given a piece of software, there is no existing technique that can predict how challenging it will be to automatically test that piece of software. In this project the goal is to develop techniques for assessing the difficulty of automatically testing software.Existing software-complexity metrics do not provide meaningful assessments of testing difficulty. This project's goal is to develop scalable techniques that can provide a quantitative assessment of testing difficulty. In order to be scalable and practical, the method has to rely on a level of abstraction that provides efficient analysis, while preserving meaningful characteristics of program behavior that relate to testing difficulty. The approach used in this project builds on two concepts that provide a promising abstraction for quantitative assessment of testing difficulty: 1) path complexity, and 2) path selectivity. Path complexity assesses how the number of paths in a given program increases with increasing execution depth, and path selectivity assesses the difficulty of finding values that satisfy a path condition. The team of researchers working on this project will develop techniques that automatically compute path complexity and path selectivity and then combine them to obtain a quantitative measure for testing difficulty. By developing techniques that can assess software-testing difficulty, this project will enable development of more effective software-testing techniques based on better resource allocation for software-quality assurance tasks. This will lead to improvements in software quality, and reduction in software defects that cause dependability and security problems. Secondly, the research activity will help to expose graduate and undergraduate students to software-quality assurance challenges and techniques. Finally, the research activity will help to disseminate the knowledge, techniques and tools developed within the scope of this project through publishing in open literature and making available the software tools that are developed as open source.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.