Free Lunch for Testing: Fuzzing Deep-Learning Libraries from Open Source
Free Lunch for Testing: Fuzzing Deep-Learning Libraries from Open Source
复制标题
DOI：
10.1145/3510003.3510041
复制
发表时间：
2022-01
期刊：
影响因子：
--
通讯作者：
Anjiang Wei;Yinlin Deng;Chenyuan Yang;Lingming Zhang
中科院分区：
文献类型：
--
作者：
Anjiang Wei;Yinlin Deng;Chenyuan Yang;Lingming Zhang
深度学习（DL）系统可以让我们的生活变得更加轻松，因此越来越受到学术界和工业界的关注。同时，DL系统中的错误可能是灾难性的，甚至可能在安全关键应用中威胁到人类生命。迄今为止，大量的研究工作一直致力于测试DL模型。然而，有趣的是，测试底层DL库的工作仍然有限，这些库是构建、优化和运行DL模型的基础。一个潜在的原因是，底层DL库的测试生成可能相当具有挑战性，因为它们的公共API主要在Python中公开，由于动态类型，甚至很难自动确定API输入参数类型。在本文中，我们提出了FreeFuzz，第一种通过从开源中挖掘来模糊DL库的方法。更具体地说，FreeFuzz从三个不同的来源获取代码/模型：1）来自库文档的代码片段，2）库开发人员测试，以及3）野生DL模型。然后，FreeFuzz自动运行所有收集的代码/模型，并使用插装来跟踪每个覆盖的API的动态信息，包括调用期间每个参数的类型和值，以及输入/输出张量的形状。最后，FreeFuzz将利用跟踪的动态信息对每个覆盖的API执行模糊测试。在PyTorch和TensorFlow这两个最流行的DL库上对FreeFuzz进行的广泛研究表明，FreeFuzz能够自动跟踪模糊1158个流行API的有效动态信息，比最先进的LEMON多9倍，开销比LEMON低3.5倍。到目前为止，FreeFuzz已经检测到PyTorch和TensorFlow的49个bug（其中38个已经被开发人员确认为以前未知的bug）。
Deep learning (DL) systems can make our life much easier, and thus are gaining more and more attention from both academia and industry. Meanwhile, bugs in DL systems can be disastrous, and can even threaten human lives in safety-critical applications. To date, a huge body of research efforts have been dedicated to testing DL models. However, interestingly, there is still limited work for testing the underlying DL libraries, which are the foundation for building, optimizing, and running DL models. One potential reason is that test generation for the underlying DL libraries can be rather challenging since their public APIs are mainly exposed in Python, making it even hard to automatically determine the API input parameter types due to dynamic typing. In this paper, we propose FreeFuzz, the first approach to fuzzing DL libraries via mining from open source. More specifically, FreeFuzz obtains code/models from three different sources: 1) code snippets from the library documentation, 2) library developer tests, and 3) DL models in the wild. Then, FreeFuzz automatically runs all the collected code/models with instrumentation to trace the dynamic information for each covered API, including the types and values of each parameter during invocation, and shapes of input/output tensors. Lastly, FreeFuzz will leverage the traced dynamic information to perform fuzz testing for each covered API. The extensive study of FreeFuzz on PyTorch and TensorFlow, two of the most popular DL libraries, shows that FreeFuzz is able to automatically trace valid dynamic information for fuzzing 1158 popular APIs, 9X more than state-of-the-art LEMON with 3.5X lower overhead than LEMON. To date, FreeFuzz has detected 49 bugs for PyTorch and TensorFlow (with 38 already confirmed by developers as previously unknown).