SaTC: CORE: Medium: Collaborative: Towards Trustworthy Deep Neural Network Based AI: A Systems Approach

人工智能（AI）有望在从技术到医学、物理学和社会科学等领域彻底改变世界。然而，随着人工智能在这些领域的部署，最近的工作表明，系统可能容易受到不同类型的攻击，导致它们行为不端;例如，导致人工智能系统将停车标志识别为限速标志的攻击。该项目旨在开发用于测试、验证和调试人工智能系统的方法，特别关注基于深度神经网络（DNN）的人工智能系统，以确保其安全性。拟议研究的智力价值包含在将开发的四个新软件工具中：（1）DeepXplore，一种用于自动和系统测试DNN的工具，可以发现可能无意或恶意引入的错误行为;（2）BadNets，一种自动生成具有已知和隐形错误行为的DNN的框架，以便对DeepXplore进行压力测试;（3）SafetyNets;一个低开销的方案，用于在云中安全和可验证地执行DNN;以及（4）VisualBackProp; DNN的可视化调试工具。将展示这些工具在自动驾驶人工智能系统安全部署中的协同使用。项目成果将显著提高人工智能系统的安全性和安全性，并增加其在安全和安全关键环境中的部署，从而产生广泛的社会影响。该项目的成果将通过出版物、讲座、开放访问代码以及Kaggle和纽约大学年度网络安全意识周（CSAW）等网站上举办的竞赛进行广泛传播。此外，将积极招募和指导科学、技术、工程和数学（STEM）领域代表性不足的少数群体学生，使其成为这一关键领域的领导者。 该项目的代码将通过github.com公开提供。即将开发的工具的初步代码已经在该网站上，包括DeepXplore（https：//github.com/peikexin9/deepxplore）和BadNets（https：//github.com/Kooscii/BadNets/）。这些存储库将从描述整个项目的主页链接到。该项目的主页将被托管在wp.nyu.edu/mlsecproject.This奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估来支持。

Artificial intelligence (AI) is poised to revolutionize the world in fields ranging from technology to medicine, physics and the social sciences. Yet as AI is deployed in these domains, recent work has shown that systems may be vulnerable to different types of attacks that cause them to misbehave; for instance, attacks that cause an AI system to recognize a stop sign as a speed-limit sign. The project seeks to develop methodologies for testing, verifying and debugging AI systems, with a specific focus on deep neural network (DNN)-based AI systems, to ensure their safety and security. The intellectual merits of the proposed research are encompassed in four new software tools that will be developed: (1) DeepXplore, a tool for automated and systematic testing of DNNs that discovers erroneous behavior that might be either inadvertently or maliciously introduced; (2) BadNets, a framework that automatically generated DNNs with known and stealthy misbehaviours in order to stress-test DeepXplore; (3) SafetyNets; a low-overhead scheme for safe and verifiable execution of DNNs in the cloud; and (4) VisualBackProp; a visual debugging tool for DNNs. The synergistic use of these tools for secure deployment of an AI system for autonomous driving will be demonstrated.The project outcomes will significantly improve the security and safety of AI systems and increase their deployment in safety- and security-critical settings, resulting in broad societal impact. The results of the project will be widely disseminated via publications, talks, open access code, and competitions hosted on sites such as Kaggle and NYU's annual Cyber-Security Awareness Week (CSAW). Furthermore, students from under-represented minority groups in science, technology, engineering and mathematics (STEM) will be actively recruited and mentored to be leaders in this critical area. The code for this project will be made publicly available via github.com. Preliminary code for the tools that will be developed is already hosted on this website, including DeepXplore (https://github.com/peikexin9/deepxplore) and BadNets (https://github.com/Kooscii/BadNets/). These repositories will be linked to from a homepage that describes the entire project. The project homepage will be hosted on wp.nyu.edu/mlsecproject.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.