SaTC: CORE: Medium: Collaborative: Towards Trustworthy Deep Neural Network Based AI: A Systems Approach

人工智能（AI）将在技术、医学、物理和社会科学等领域彻底改变世界。然而，随着人工智能在这些领域的应用，最近的研究表明，系统可能容易受到不同类型的攻击，导致它们行为不端；例如，导致人工智能系统将停车标志识别为限速标志的攻击。该项目旨在开发测试、验证和调试人工智能系统的方法，特别关注基于深度神经网络（DNN）的人工智能系统，以确保其安全性。拟议研究的知识价值包含在将开发的四个新软件工具中：(1)DeepXplore，这是一个用于自动和系统测试深度神经网络的工具，可以发现可能是无意或恶意引入的错误行为；(2) BadNets，一个自动生成具有已知和隐形错误行为的深度神经网络的框架，用于对DeepXplore进行压力测试；(3)社保网;在云中安全且可验证地执行dnn的低开销方案；(4) VisualBackProp；dnn的可视化调试工具。将展示如何协同使用这些工具来安全部署用于自动驾驶的人工智能系统。该项目成果将显著提高人工智能系统的安全性，并增加其在安全和安全关键环境中的部署，从而产生广泛的社会影响。该项目的成果将通过出版物、讲座、开放获取代码和在Kaggle和纽约大学年度网络安全意识周（CSAW）等网站上举办的竞赛等方式广泛传播。此外，在科学、技术、工程和数学（STEM）领域，来自代表性不足的少数群体的学生将被积极招募和指导，成为这一关键领域的领导者。该项目的代码将通过github.com公开提供。将开发的工具的初步代码已经托管在这个网站上，包括DeepXplore （https://github.com/peikexin9/deepxplore）和BadNets （https://github.com/Kooscii/BadNets/）。这些存储库将从描述整个项目的主页链接到。该项目的主页将放在wp.nyu.edu/mlsecproject.This上，该奖项反映了美国国家科学基金会的法定使命，并通过基金会的智力价值和更广泛的影响审查标准进行评估，认为值得支持。

Artificial intelligence (AI) is poised to revolutionize the world in fields ranging from technology to medicine, physics and the social sciences. Yet as AI is deployed in these domains, recent work has shown that systems may be vulnerable to different types of attacks that cause them to misbehave; for instance, attacks that cause an AI system to recognize a stop sign as a speed-limit sign. The project seeks to develop methodologies for testing, verifying and debugging AI systems, with a specific focus on deep neural network (DNN)-based AI systems, to ensure their safety and security. The intellectual merits of the proposed research are encompassed in four new software tools that will be developed: (1) DeepXplore, a tool for automated and systematic testing of DNNs that discovers erroneous behavior that might be either inadvertently or maliciously introduced; (2) BadNets, a framework that automatically generated DNNs with known and stealthy misbehaviours in order to stress-test DeepXplore; (3) SafetyNets; a low-overhead scheme for safe and verifiable execution of DNNs in the cloud; and (4) VisualBackProp; a visual debugging tool for DNNs. The synergistic use of these tools for secure deployment of an AI system for autonomous driving will be demonstrated.The project outcomes will significantly improve the security and safety of AI systems and increase their deployment in safety- and security-critical settings, resulting in broad societal impact. The results of the project will be widely disseminated via publications, talks, open access code, and competitions hosted on sites such as Kaggle and NYU's annual Cyber-Security Awareness Week (CSAW). Furthermore, students from under-represented minority groups in science, technology, engineering and mathematics (STEM) will be actively recruited and mentored to be leaders in this critical area. The code for this project will be made publicly available via github.com. Preliminary code for the tools that will be developed is already hosted on this website, including DeepXplore (https://github.com/peikexin9/deepxplore) and BadNets (https://github.com/Kooscii/BadNets/). These repositories will be linked to from a homepage that describes the entire project. The project homepage will be hosted on wp.nyu.edu/mlsecproject.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.