Distribution-Aware Testing of Neural Networks Using Generative Models
Distribution-Aware Testing of Neural Networks Using Generative Models
复制标题
DOI：
10.1109/icse43902.2021.00032
复制
发表时间：
2021-02
期刊：
影响因子：
--
通讯作者：
Swaroopa Dola;Matthew B. Dwyer;M. Soffa
中科院分区：
文献类型：
--
作者：
Swaroopa Dola;Matthew B. Dwyer;M. Soffa
由于越来越多的关键应用程序部署了深度神经网络（DNN），因此具有深度神经网络（DNN）作为组件的软件的可靠性在今天变得非常重要。对可靠性的需求提出了对这些系统的安全性和可靠性进行严格测试的需要。在过去的几年里，有许多研究工作集中在测试DNN上。然而，目前提出的测试生成技术缺乏检查，以确定他们正在生成的测试输入是否有效，从而产生无效的输入。为了说明这种情况，我们探讨了三种最近的DNN测试技术。使用基于深度生成模型的输入验证，我们证明了所有这三种技术都会生成大量的无效测试输入。我们进一步分析了DNN测试技术生成的测试输入所实现的测试覆盖率，并展示了无效的测试输入如何错误地夸大测试覆盖率指标。为了克服测试中包含无效输入的问题，我们提出了一种技术，将测试中的DNN模型的有效输入空间纳入测试生成过程。我们的技术使用基于模型的深度生成算法来仅生成有效输入。我们的实证研究结果表明，我们的技术是有效的，在消除无效的测试和提高有效的测试输入生成的数量。
The reliability of software that has a Deep Neural Network (DNN) as a component is urgently important today given the increasing number of critical applications being deployed with DNNs. The need for reliability raises a need for rigorous testing of the safety and trustworthiness of these systems. In the last few years, there have been a number of research efforts focused on testing DNNs. However the test generation techniques proposed so far lack a check to determine whether the test inputs they are generating are valid, and thus invalid inputs are produced. To illustrate this situation, we explored three recent DNN testing techniques. Using deep generative model based input validation, we show that all the three techniques generate significant number of invalid test inputs. We further analyzed the test coverage achieved by the test inputs generated by the DNN testing techniques and showed how invalid test inputs can falsely inflate test coverage metrics. To overcome the inclusion of invalid inputs in testing, we propose a technique to incorporate the valid input space of the DNN model under test in the test generation process. Our technique uses a deep generative model-based algorithm to generate only valid inputs. Results of our empirical studies show that our technique is effective in eliminating invalid tests and boosting the number of valid test inputs generated.