Collaborative Research: SHF: Small: Automated Quantitative Assessment of Testing Difficulty

我们的社会严重依赖于我们周围越来越多的可编程设备上运行的软件系统。此外，汽车和飞机等安全关键系统中的软件数量不断增加。软件质量保证是我们在这个现代计算主导的时代面临的最基本的问题之一。人们每天都可以在新闻中看到由劣质软件引起的可靠性和安全问题。开发能够提高软件系统质量的技术，避免软件系统在运行过程中造成灾难性后果，这一点极其重要。最常见的软件质量保证技术是软件测试。尽管自动化软件测试技术取得了巨大进步，但很难预测其有效性。给定一个软件，没有现有技术可以预测自动测试该软件的挑战性。在该项目中，目标是开发用于评估自动测试软件难度的技术。现有的软件复杂性指标无法提供有意义的测试难度评估。该项目的目标是开发可扩展的技术，可以提供测试难度的定量评估。为了可扩展和实用，该方法必须依赖于提供有效分析的抽象级别，同时保留与测试难度相关的程序行为的有意义的特征。该项目中使用的方法基于两个概念，为测试难度的定量评估提供了有前途的抽象：1）路径复杂性，2）路径选择性。路径复杂性评估给定程序中的路径数量如何随着执行深度的增加而增加，路径选择性评估找到满足路径条件的值的难度。从事该项目的研究人员团队将开发自动计算路径复杂性和路径选择性的技术，然后将它们结合起来以获得测试难度的定量度量。通过开发可以评估软件测试难度的技术，该项目将能够基于更好的软件质量保证任务资源分配来开发更有效的软件测试技术。这将提高软件质量，并减少导致可靠性和安全问题的软件缺陷。其次，研究活动将有助于让研究生和本科生接触软件质量保证挑战和技术。最后，研究活动将通过公开文献发表并提供开源开发的软件工具，帮助传播本项目范围内开发的知识、技术和工具。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Our society is heavily reliant on software systems running on an increasingly large number of programmable devices that surround us. Moreover, the amount of software in safety-critical systems such as cars and planes keeps increasing. Software-quality assurance is one of the most fundamental problems that we are facing in this modern computing-dominated era. One can read about dependability and security problems caused by poor-quality software in the news everyday. It is extremely crucial to develop techniques that can improve the quality of software systems before they cause disastrous consequences during operation. The most common software-quality assurance technique is software testing. Although there has been a surge of progress in automated software-testing techniques, it is hard to predict their effectiveness. Given a piece of software, there is no existing technique that can predict how challenging it will be to automatically test that piece of software. In this project the goal is to develop techniques for assessing the difficulty of automatically testing software.Existing software-complexity metrics do not provide meaningful assessments of testing difficulty. This project's goal is to develop scalable techniques that can provide a quantitative assessment of testing difficulty. In order to be scalable and practical, the method has to rely on a level of abstraction that provides efficient analysis, while preserving meaningful characteristics of program behavior that relate to testing difficulty. The approach used in this project builds on two concepts that provide a promising abstraction for quantitative assessment of testing difficulty: 1) path complexity, and 2) path selectivity. Path complexity assesses how the number of paths in a given program increases with increasing execution depth, and path selectivity assesses the difficulty of finding values that satisfy a path condition. The team of researchers working on this project will develop techniques that automatically compute path complexity and path selectivity and then combine them to obtain a quantitative measure for testing difficulty. By developing techniques that can assess software-testing difficulty, this project will enable development of more effective software-testing techniques based on better resource allocation for software-quality assurance tasks. This will lead to improvements in software quality, and reduction in software defects that cause dependability and security problems. Secondly, the research activity will help to expose graduate and undergraduate students to software-quality assurance challenges and techniques. Finally, the research activity will help to disseminate the knowledge, techniques and tools developed within the scope of this project through publishing in open literature and making available the software tools that are developed as open source.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.