EAGER: SaTC-EDU: Multi-Level Attack and Defense Simulation Environment for Artificial Intelligence Education and Research

人工智能（AI）技术，特别是机器学习（ML），越来越多地集成到安全和安全关键型应用中，如自动驾驶汽车和恶意软件检测。然而，研究表明，人工智能技术可能容易受到网络攻击，如对抗性干扰和数据中毒，当人工智能系统做出的决策被操纵时，可能会导致灾难性的后果。尽管在这一领域进行了大量的研究工作，但研究界不成比例地集中在少数几个领域，如图像识别和一些简单的对抗性设置。与此同时，更多的安全关键领域，如恶意软件检测，以及更全面地代表现实世界的各种对抗模型，都被忽视了。此外，很难比较、对比和描述开发强大人工智能系统的不同方法，因为这一领域的工作是分散的。这也为人工智能和网络安全的教育工作带来了挑战。该项目旨在通过人工智能、网络安全和教育方面的协同努力来解决这些紧迫问题，这些努力将产生重大的研究和社会影响。首先，该项目的结果将通过传播工具和材料，促进公众对鲁棒性AI问题和研究的认识。其次，该项目将把强大人工智能的研究进展民主化到目前服务不足的应用领域，例如恶意软件检测。第三，该项目代表了培养具有构建强大和安全的人工智能系统技能的劳动力的具体步骤。该项目开发的平台将被纳入加州尔湾大学的本科生和研究生课程，并向研究人员和教育工作者公开提供。该项目的具体目标是解决强大和安全的人工智能中的研究碎片问题。该项目团队将开发一个名为Maestro的新平台，用于模拟对抗性机器学习任务，涵盖各种对抗性能力（获取梯度、模型权重、预测等）。在正式的访问控制框架下进行攻击和防御。Maestro平台将使其更容易实现，比较和开发尚未充分探索的新型对抗性ML算法，设置和应用程序，包括自然语言处理（NLP）的后门利用，隐形对抗性恶意软件生成和程序嵌入的安全分析。Maestro的架构不仅提供了一个有用的框架来构建人工智能和网络安全的教学材料，还将用于使用主动学习和游戏化策略构建课程材料。 后者将吸引学生，同时教授他们关于构建可靠和强大的人工智能系统的基本概念。该项目由安全和可信赖的网络空间（SaTC）计划的一项特别倡议支持，旨在促进网络安全，人工智能和教育领域之间的新的，以前未探索的合作。SATC计划与联邦网络安全研究和发展战略计划和国家隐私研究战略保持一致，以保护和维护网络系统日益增长的社会和经济效益，同时确保安全和隐私。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Artificial intelligence (AI) techniques, particularly machine learning (ML), are increasingly integrated into safety- and security-critical applications such as autonomous vehicles and malware detection. However, research has shown AI techniques can be vulnerable to cyber-attacks such as adversarial perturbation and data poisoning, potentially leading to catastrophic outcomes when decisions made by AI systems are manipulated. Despite significant research efforts in this area, the research community has disproportionately focused on only a few domains, such as image recognition, and a few simple adversarial setups. Meanwhile more security-critical domains, such as malware detection, and a variety of adversarial models that more fully represent the real-world, have been ignored. Furthermore, it is difficult to compare, contrast, and characterize the different approaches to developing robust AI systems because of the fragmented nature of efforts in this area. This also creates challenges for education efforts in AI and cybersecurity. This project aims to address these urgent issues with synergistic efforts in AI, cybersecurity, and education that will produce significant research and societal impacts. First, the results of the project will promote public awareness of the issues and research around the robustness AI via the dissemination of tools and materials. Second, the project will democratize research progress in robust AI to application domains that are currently underserved, such as malware detection. Third, the project represents a concrete step towards fostering a workforce with skills in building robust and secure AI systems. The platform developed by this project will be integrated into undergraduate and graduate courses at the University of California Irvine and made publicly available to researchers and educators. The specific aim of the project is to address issues of research fragmentation in robust and secure AI. The project team will develop a new platform, called Maestro, to simulate adversarial machine learning tasks, covering a variety of adversarial capabilities (access to gradients, model weights, predictions, etc.) for both attacks and defenses, under a formal access-control framework. The Maestro platform will make it easier to implement, compare, and develop novel adversarial ML algorithms, settings, and applications that have not been sufficiently explored, including backdoor exploitation of natural language processing (NLP), stealthy adversarial malware generation, and security analysis of program embedding. The architecture of Maestro not only provides a useful framework to structure pedagogical materials in AI and cybersecurity, but also will be used to build course materials using active learning and gamification strategies. The latter will engage students while teaching them essential concepts about building reliable and robust AI systems.This project is supported by a special initiative of the Secure and Trustworthy Cyberspace (SaTC) program to foster new, previously unexplored, collaborations between the fields of cybersecurity, artificial intelligence, and education. The SaTC program aligns with the Federal Cybersecurity Research and Development Strategic Plan and the National Privacy Research Strategy to protect and preserve the growing social and economic benefits of cyber systems while ensuring security and privacy.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.