CEDAR: Continuous Testing of Deep Learning Libraries
CEDAR: Continuous Testing of Deep Learning Libraries
复制标题
DOI：
--
复制
发表时间：
--
期刊：
影响因子：
--
通讯作者：
Danning Xie;Jiannan Wang;†. HungVietPham;Lin Tan;Yu Guo;Adnan Aziz;Erik Meijer
中科院分区：
文献类型：
--
作者：
Danning Xie;Jiannan Wang;†. HungVietPham;Lin Tan;Yu Guo;Adnan Aziz;Erik Meijer
由于深度学习（DL）库经历了快速的开发，每天有数千行代码更改，因此需要持续测试以检测软件错误并确保代码质量。在本文中，我们探讨了在连续测试环境中的DL测试方法。为了使其可行，我们提出了第一个用于DL库的持续测试框架- CEDAR -它集成了两种最先进的DL测试方法（DocTer和EAGLE），有效地测试了两种流行的DL库PyTorch和TensorFlow。通过将CEDAR应用于20个版本的PyTorch和TensorFlow，CEDAR在140个API中检测到83个bug。在这83个漏洞中，有23个是以前未知的漏洞，其中21个已被开发人员确认或修复。结果还显示，CEDAR有效地缩短了bug检测延迟，平均缩短了近一年（338.6天）。此外，CEDAR证明了它在检测新的回归错误和隐藏错误方面的有效性。通过三种优化策略，CEDAR将时间和空间开销分别降低了15.4和9.7倍。我们分享从我们的研究中获得的见解和经验教训，旨在推动更有效和高效的DL库持续测试的发展，贝内开发人员和研究人员受益。
—Since Deep Learning (DL) libraries undergo rapid development with thousands of lines of code changes daily, they require continuous testing to detect software bugs and ensure code quality. In this paper, we explore DL testing approaches in a continuous testing setting. To make it feasible, we present the first continuous testing framework for DL libraries— CEDAR —that integrates two state-of-the-art DL testing approaches (DocTer and EAGLE) efficiently to test two popular DL libraries, PyTorch and TensorFlow. Through the application of CEDAR to 20 versions of PyTorch and TensorFlow, CEDAR detects 83 bugs in 140 APIs. Out of the 83 bugs, 23 are previously unknown bugs with 21 confirmed or fixed by the developers. The results also show CEDAR has effectively shortened the bug detection latency by almost a year (338.6 days) on average. In addition, CEDAR demonstrates its effectiveness in detecting new regression bugs and masked bugs. With three optimization strategies, CEDAR reduces the time and space overhead by a factor of 15.4 and 9.7. We share insights and lessons learned from our research, aiming to advance the development of more effective and efficient continuous testing for DL libraries, benefiting both developers and researchers.