SHF: Small: Distribution-aware Testing for Neural Networks

机器学习旨在从一组示例中学习软件实现的定义。 例如，从来自汽车中的前向摄像头的大量图像中，这种方法可以学习检测行人何时靠近道路。 这种方法很有吸引力，因为传统的软件开发方法很难处理这种问题规范-考虑到精确描述行人的外观的挑战，考虑到他们的衣服，他们携带的东西，他们的位置和姿势的所有变化，他们是否部分被遮挡，部分被照亮等。因此，可以使用新的能力来部署可能更好的系统。 挑战在于如何确定这些新功能是否按预期工作。 对于传统软件，数十年的研究已经产生了用于测试软件的复杂框架。 该项目将采用该研究机构的策略来测试机器学习模型。 通过这种方式，该项目旨在提高基于机器学习的高级软件系统的安全性和质量，这将对依赖这些系统的人和组织产生积极影响。该项目将利用目标数据分布的学习模型来驱动对该分布样本进行训练的神经网络的测试。 所开发的方法将利用生成模型，这些模型表现出高精度和目标分布的召回率，并且具有对其潜在空间定义良好的数学结构。 通过瞄准这个潜在空间，分布感知神经网络测试技术可以利用其相对于神经网络输入域的降维来评估给定测试套件的充分性并生成新的有价值的测试。 此外，该项目将开发一个严格的基于突变的框架，用于评估神经网络测试方法的故障检测有效性，并将该框架用于评估分布感知测试方法的好处。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

Machine learning aims to learn the definition of a software implementation from a set of examples. For instance, from a large set of images from the forward facing camera in an automobile such approaches can learn to detect when a pedestrian is near the road. This method is attractive because traditional software development approaches have a hard time with this kind of problem specification -- consider the challenge of precisely describing what a pedestrian looks like given all the variations in their clothing, what they are carrying, their location and pose, whether they are partially obscured, partially lit, etc. Using machine learning, new capabilities can thus be used to deploy potentially better systems. The challenge is how to determine that these new capabilities work as expected. For traditional software, decades of research has yielded sophisticated frameworks for testing software. This project will adapt the strategies from that body of research to test machine learning models. In this way the project aims to improve the safety and quality of advanced machine-learning-based software systems, which will have positive impacts on people and organizations who depend on those systems.The project will exploit learned models of a target data distribution to drive testing of neural networks that are trained on samples from that distribution. The methods that are developed will exploit generative models that exhibit high-precision and recall of the target distribution and that have a well-defined mathematical structure to their latent space. By targeting this latent space, distribution-aware neural-network testing techniques can leverage its reduced dimensionality, relative to the neural network's input domain, to assess the adequacy of a given test suite and to generate new and valuable tests. In addition, the project will develop a rigorous mutation-based framework for evaluating the fault-detection effectiveness of neural network testing methods and to employ that framework in evaluating the benefits of distribution-aware testing methods.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.