SaTC: CORE: Medium: Hidden Rules in Neural Networks as Attacks and Adversarial Defenses

深度神经网络（dnn）的最新进展使语音/面部识别、语言翻译和图像识别等技术挑战取得了重大进展。然而，dnn仍然容易受到一类被称为“后门”或“特洛伊”攻击的隐藏攻击的攻击，在这些攻击中，隐藏规则被训练成一个模型，该模型只在具有一些不寻常属性的模型输入上变得活跃，包括一个“触发器”。它们足够强大，以至于一个小而不显眼的触发因素就能使模型产生意想不到的（通常是错误的）结果，例如，将任何有黑色十字章纹身的人识别为预先确定的名人。尽管最近做出了一些努力，但人们对这些攻击仍然知之甚少，而且强大的防御仍然难以捉摸。本项目深入研究这类攻击，以了解它们对真实机器学习系统和潜在防御的潜在影响。更具体地说，该项目将首先对跨多个领域（和潜在防御）的后门攻击的广度进行分类，包括图像（面部和物体识别）、文本（自然语言处理和情感分析）和音频（说话人识别和语音转录）。然后，该项目将探索它们在数字领域之外的实际意义，包括物理世界中的后门攻击（如面部识别），以及与迁移学习共存的高级后门，迁移学习是当今共享DNN模型的流行方法。最后，该项目将探索后门作为模型训练工具的潜在积极用途，通过使用后门技术构建的蜜罐捕获对抗性攻击，为DNN模型产生一种新的保护机制。这些技术将在广泛的应用、数据集和模型中结合对高级攻击和防御的评估，并在可能的情况下，在物理领域进行实验。这个项目的成功结果应该提醒安全专业人员注意dnn中后门的风险，同时提供强大防御所需的软件和算法工具。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Recent advances in Deep Neural Networks (DNNs) have enabled significant progress in technological challenges such as voice/facial recognition, language translation and image recognition. Yet DNNs remain vulnerable to a class of hidden attacks called "backdoor" or "Trojan" attacks, where hidden rules are trained into a model which only become active on model input with some unusual properties, comprising a "trigger." They are strong enough that the presence of a small, inconspicuous trigger can make the model produce unexpected (and often erroneous) results, e.g., recognize anyone with a black ankh tattoo as a predetermined celebrity. Despite recent efforts, these attacks remain poorly understood, and robust defenses remain elusive. This project studies this class of attacks in depth to understand their potential impact on real machine learning systems and potential defenses.More specifically, the project will first catalog the breadth of backdoor attacks across multiple domains (and potential defenses), including images (facial and object recognition), text (natural language processing and sentiment analysis), and audio (speaker recognition and voice transcription). The project will then explore their practical implications outside the digital domain, including backdoor attacks in the physical world (such as on facial recognition), and advanced backdoors that coexist with transfer learning, the prevailing method for sharing DNN models today. Finally, the project will explore potential positive uses of backdoors as model-training tools, spawning a novel protection mechanism for DNN models, by trapping adversarial attacks with honey-pots built using backdoor techniques. The techniques will incorporate evaluation of both advanced attacks and defenses across a broad range of applications, datasets and models, and whenever possible, experiments in the physical domain. Successful results from this project should alert security professionals to the risk of backdoors in DNNs, while providing the software and algorithmic tools necessary for robust defenses.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.