SHF: Small: Distribution-aware Testing for Neural Networks

SHF：小型：神经网络的分布感知测试

• 批准号: 2129824
• 负责人: Matthew Dwyer
• 金额: $ 49.85万
• 依托单位: University of Virginia Main Campus
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2129824&HistoricalAwards=false
• 关键词: SHF; Small; Distribution; aware; Testing

Machine learning aims to learn the definition of a software implementation from a set of examples. For instance, from a large set of images from the forward facing camera in an automobile such approaches can learn to detect when a pedestrian is near the road. This method is attractive because traditional software development approaches have a hard time with this kind of problem specification -- consider the challenge of precisely describing what a pedestrian looks like given all the variations in their clothing, what they are carrying, their location and pose, whether they are partially obscured, partially lit, etc. Using machine learning, new capabilities can thus be used to deploy potentially better systems. The challenge is how to determine that these new capabilities work as expected. For traditional software, decades of research has yielded sophisticated frameworks for testing software. This project will adapt the strategies from that body of research to test machine learning models. In this way the project aims to improve the safety and quality of advanced machine-learning-based software systems, which will have positive impacts on people and organizations who depend on those systems.The project will exploit learned models of a target data distribution to drive testing of neural networks that are trained on samples from that distribution. The methods that are developed will exploit generative models that exhibit high-precision and recall of the target distribution and that have a well-defined mathematical structure to their latent space. By targeting this latent space, distribution-aware neural-network testing techniques can leverage its reduced dimensionality, relative to the neural network's input domain, to assess the adequacy of a given test suite and to generate new and valuable tests. In addition, the project will develop a rigorous mutation-based framework for evaluating the fault-detection effectiveness of neural network testing methods and to employ that framework in evaluating the benefits of distribution-aware testing methods.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

机器学习的目的是从一组例子中学习软件实现的定义。例如，从汽车前置摄像头的一大组图像中，这种方法可以学习检测行人何时靠近道路。这种方法很有吸引力，因为传统的软件开发方法很难处理这种问题说明--考虑到在给定行人衣服的所有变化、他们携带的东西、他们的位置和姿势、他们是否部分被遮挡、部分被照亮等情况下，准确描述行人的样子是一项挑战。使用机器学习，因此可以使用新的功能来部署潜在更好的系统。挑战在于如何确定这些新功能是否按预期工作。对于传统软件，几十年的研究已经产生了复杂的软件测试框架。这个项目将调整该研究机构的策略，以测试机器学习模型。通过这种方式，该项目旨在提高基于机器学习的先进软件系统的安全性和质量，这将对依赖这些系统的个人和组织产生积极影响。该项目将利用目标数据分布的学习模型来驱动对神经网络的测试，这些神经网络是根据该分布的样本进行训练的。所开发的方法将利用生成性模型，这些模型表现出对目标分布的高精度和回忆性，并且具有对其潜在空间的明确的数学结构。通过瞄准这一潜在空间，分布感知神经网络测试技术可以利用其相对于神经网络输入域的降维来评估给定测试套件的充分性，并生成新的有价值的测试。此外，该项目将开发一个严格的基于突变的框架来评估神经网络测试方法的故障检测有效性，并使用该框架来评估分布感知测试方法的好处。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Input Distribution Coverage: Measuring Feature Interaction Adequacy in Neural Network Testing

• DOI: 10.1145/3576040
• 发表时间: 2022-12
• 期刊: ACM Transactions on Software Engineering and Methodology
• 影响因子: 4.4
• 作者: Swaroopa Dola;Matthew B. Dwyer;M. Soffa