EAGER: SaTC-EDU: Multi-Level Attack and Defense Simulation Environment for Artificial Intelligence Education and Research

人工智能（AI）技术，特别是机器学习（ML），越来越多地集成到安全关键应用中，如自动驾驶汽车和恶意软件检测。然而，研究表明，人工智能技术很容易受到网络攻击，比如对抗性扰动和数据中毒，当人工智能系统做出的决定被操纵时，可能会导致灾难性的后果。尽管在这一领域进行了大量的研究工作，但研究界不成比例地只关注少数领域，如图像识别和一些简单的对抗性设置。与此同时，更多的安全关键领域，如恶意软件检测，以及各种更全面地代表现实世界的对抗模型，都被忽视了。此外，很难比较、对比和描述开发健壮的人工智能系统的不同方法，因为这一领域的工作是分散的。这也给人工智能和网络安全方面的教育工作带来了挑战。该项目旨在通过人工智能、网络安全和教育领域的协同努力解决这些紧迫问题，从而产生重大的研究和社会影响。首先，该项目的成果将通过工具和材料的传播，促进公众对鲁棒性人工智能的问题和研究的认识。其次，该项目将使健壮人工智能的研究进展民主化，以应用于目前服务不足的应用领域，如恶意软件检测。第三，该项目是朝着培养具有构建强大和安全人工智能系统技能的劳动力迈出的具体一步。该项目开发的平台将被整合到加州大学欧文分校的本科和研究生课程中，并向研究人员和教育工作者公开提供。该项目的具体目标是解决强大和安全的人工智能研究碎片化的问题。项目团队将开发一个名为Maestro的新平台，以模拟对抗性机器学习任务，在正式的访问控制框架下，涵盖攻击和防御的各种对抗性功能（访问梯度、模型权重、预测等）。Maestro平台将更容易实现、比较和开发尚未充分探索的新型对抗性ML算法、设置和应用程序，包括自然语言处理（NLP）的后门利用、隐形对抗性恶意软件生成和程序嵌入的安全分析。Maestro的架构不仅为构建人工智能和网络安全的教学材料提供了有用的框架，而且还将用于使用主动学习和游戏化策略构建课程材料。后者将吸引学生，同时教授他们构建可靠和强大的人工智能系统的基本概念。该项目由安全与可信网络空间（SaTC）计划的一项特别倡议支持，旨在促进网络安全、人工智能和教育领域之间前所未有的合作。SaTC项目与《联邦网络安全研究与发展战略计划》和《国家隐私研究战略》保持一致，旨在保护和维护网络系统日益增长的社会和经济效益，同时确保安全和隐私。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Artificial intelligence (AI) techniques, particularly machine learning (ML), are increasingly integrated into safety- and security-critical applications such as autonomous vehicles and malware detection. However, research has shown AI techniques can be vulnerable to cyber-attacks such as adversarial perturbation and data poisoning, potentially leading to catastrophic outcomes when decisions made by AI systems are manipulated. Despite significant research efforts in this area, the research community has disproportionately focused on only a few domains, such as image recognition, and a few simple adversarial setups. Meanwhile more security-critical domains, such as malware detection, and a variety of adversarial models that more fully represent the real-world, have been ignored. Furthermore, it is difficult to compare, contrast, and characterize the different approaches to developing robust AI systems because of the fragmented nature of efforts in this area. This also creates challenges for education efforts in AI and cybersecurity. This project aims to address these urgent issues with synergistic efforts in AI, cybersecurity, and education that will produce significant research and societal impacts. First, the results of the project will promote public awareness of the issues and research around the robustness AI via the dissemination of tools and materials. Second, the project will democratize research progress in robust AI to application domains that are currently underserved, such as malware detection. Third, the project represents a concrete step towards fostering a workforce with skills in building robust and secure AI systems. The platform developed by this project will be integrated into undergraduate and graduate courses at the University of California Irvine and made publicly available to researchers and educators. The specific aim of the project is to address issues of research fragmentation in robust and secure AI. The project team will develop a new platform, called Maestro, to simulate adversarial machine learning tasks, covering a variety of adversarial capabilities (access to gradients, model weights, predictions, etc.) for both attacks and defenses, under a formal access-control framework. The Maestro platform will make it easier to implement, compare, and develop novel adversarial ML algorithms, settings, and applications that have not been sufficiently explored, including backdoor exploitation of natural language processing (NLP), stealthy adversarial malware generation, and security analysis of program embedding. The architecture of Maestro not only provides a useful framework to structure pedagogical materials in AI and cybersecurity, but also will be used to build course materials using active learning and gamification strategies. The latter will engage students while teaching them essential concepts about building reliable and robust AI systems.This project is supported by a special initiative of the Secure and Trustworthy Cyberspace (SaTC) program to foster new, previously unexplored, collaborations between the fields of cybersecurity, artificial intelligence, and education. The SaTC program aligns with the Federal Cybersecurity Research and Development Strategic Plan and the National Privacy Research Strategy to protect and preserve the growing social and economic benefits of cyber systems while ensuring security and privacy.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.