DeepHunter: a coverage-guided fuzz testing framework for deep neural networks
DeepHunter: a coverage-guided fuzz testing framework for deep neural networks
复制标题
DOI：
10.1145/3293882.3330579
复制
发表时间：
2019-07
期刊：
影响因子：
--
通讯作者：
Xiaofei Xie;L. Ma;Felix Juefei-Xu;Minhui Xue;Hongxu Chen;Yang Liu;Jianjun Zhao;Bo Li;
中科院分区：
文献类型：
--
作者：
Xiaofei Xie;L. Ma;Felix Juefei-Xu;Minhui Xue;Hongxu Chen;Yang Liu;Jianjun Zhao;Bo Li;
在过去的十年中，我们看到了将基于深度神经网络（DNN）的软件应用于安全关键场景（如自动驾驶）的巨大潜力。与传统软件类似，DNN可能会表现出由隐藏缺陷引起的错误行为，导致严重的事故和损失。在本文中，我们提出了DeepHunter，这是一个覆盖引导的模糊测试框架，用于检测通用DNN的潜在缺陷。为此，我们首先提出了一种变形变异策略来生成新的语义保留的测试，并利用多个可扩展的覆盖标准作为反馈来指导测试生成。我们进一步提出了一个种子选择策略，结合了基于多样性和基于最近的种子选择。我们在DeepHunter中实施并整合了5个现有的测试标准和4个种子选择策略。大规模的实验表明：（1）我们的变形变异策略可以有效地生成与原始种子语义相同的新的有效测试，有效率高达98%;（2）基于多样性的种子选择在提高覆盖率和检测缺陷方面比基于最近度的种子选择更重要;（3）DeepHunter在覆盖率以及识别出的缺陷的数量和多样性方面优于最先进的技术;（4）在基于角区域的标准的指导下，DeepHunter有助于在DNN量化期间捕获平台迁移的缺陷。
The past decade has seen the great potential of applying deep neural network (DNN) based software to safety-critical scenarios, such as autonomous driving. Similar to traditional software, DNNs could exhibit incorrect behaviors, caused by hidden defects, leading to severe accidents and losses. In this paper, we propose DeepHunter, a coverage-guided fuzz testing framework for detecting potential defects of general-purpose DNNs. To this end, we first propose a metamorphic mutation strategy to generate new semantically preserved tests, and leverage multiple extensible coverage criteria as feedback to guide the test generation. We further propose a seed selection strategy that combines both diversity-based and recency-based seed selection. We implement and incorporate 5 existing testing criteria and 4 seed selection strategies in DeepHunter. Large-scale experiments demonstrate that (1) our metamorphic mutation strategy is useful to generate new valid tests with the same semantics as the original seed, by up to a 98% validity ratio; (2) the diversity-based seed selection generally weighs more than recency-based seed selection in boosting the coverage and in detecting defects; (3) DeepHunter outperforms the state of the arts by coverage as well as the quantity and diversity of defects identified; (4) guided by corner-region based criteria, DeepHunter is useful to capture defects during the DNN quantization for platform migration.