CAREER: Systematic Software Testing for Deep Learning Applications

软件开发的模式正在发生转变，决策越来越多地从手工编码的程序逻辑转移到依赖深度学习(DL)-语音处理、图像识别、机器人等流行的应用程序正在使用深度学习来实现其核心组件。深度神经网络(DNNS)是一种广泛使用的深度神经网络，是这一进步背后的关键。随着传统应用的如此惊人的增长，DNN和其他DL技术也越来越多地被用于安全关键型系统，如自动驾驶汽车、医疗诊断、恶意软件检测和飞机防撞系统。如此广泛地采用DL技术伴随着对这些系统的可靠性的担忧，因为已经报道了几个基于DL的行为的高调实例。因此，使用逼真的转角情况严格测试这些应用程序以确保高可靠性就变得至关重要。然而，由于诸如DNN之类的DL实现与传统软件之间的基本体系结构差异，现有的软件测试技术并不以任何明显的方式适用于它们。事实上，谷歌、特斯拉等公司正面临着越来越多的软件测试挑战，以确保可靠和安全的数字图书馆应用程序。因此，系统地测试基于数字图书馆的软件系统将是提高敏感和安全关键数字图书馆系统的安全性和可靠性的重要一步。本项目将设计、实现和评估一个新的软件测试框架，以评估深度学习应用程序的可靠性，并在应用程序开发和维护阶段检测错误行为。特别是，建议的框架将开发新的白盒测试策略、现实的测试用例生成技术和回归测试技术来评估DL应用程序。基于DL的编程范例的一个独特特征是最终应用程序高度依赖训练数据。因此，该研究将构建新的白盒测试策略，将模型和训练数据作为一个整体进行评估。此外，这项研究将设计和部署技术来生成新的测试用例，这些测试用例可以捕获DL应用程序可能失败的真实世界的角例行为。该项目还将调查数据或模型架构中的任何更改如何影响预先训练的模型，以指导回归测试用例选择和优先处理过程。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

A paradigm shift is underway in software development, where decision making is increasingly shifting from hand-coded program logic to a reliance on Deep Learning (DL) --- popular applications of Speech Processing, Image Recognition, Robotics, etc. are using DL to implement their core components. Deep Neural Networks (DNNs), a widely used form of DL, is a key behind much of this progress. With such spectacular growth in traditional applications, DNNs and other DL technologies are also increasingly being used in safety-critical systems such as autonomous cars, medical diagnosis, malware detection, and aircraft collision avoidance systems. Such a wide adoption of DL techniques carries with it concerns about the reliability of these systems, as several high-profile instances of DL-based behavior have already been reported. Thus, it has become crucial to rigorously test these applications with realistic corner cases to ensure high reliability. However, due to the fundamental architectural differences between DL implementations such as DNNs and traditional software, existing software testing techniques do not apply to them in any obvious way. In fact, companies like Google, Tesla, etc. are increasingly confronting software testing challenges to ensure reliable and safe DL applications. Therefore, systematically testing DL-based software systems will be a significant step towards increasing safety and reliability of sensitive and safety-critical DL systems.This project will design, implement, and evaluate a novel software testing framework to assess the reliability of the Deep Learning applications and detect buggy behaviors during the application development and maintenance phase. In particular, the proposed framework will develop novel white-box testing strategies, realistic test-case generation techniques, and regression testing techniques to assess DL applications. A unique characteristic of the DL-based programming paradigm is that the end applications highly depend on the training data. Therefore, the research will build novel white-box testing strategies to evaluate both the model and the training data together as a whole system. In addition, this research will design and deploy techniques to generate new test cases that capture the real-world corner-case behavior where the DL applications may fail. The project will also investigate how any changes in data or model architecture can impact a pre-trained model in order to guide regression test case selection and prioritization process.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.