CAREER: Learning to Secure Cooperative Multi-Agent Learning Systems: Advanced Attacks and Robust Defenses

协作多智能体学习（MAL），其中多个智能体学习相互协调和与人类协调，正在成为解决各种安全和安全关键领域（包括运输，电力系统，机器人和医疗保健）中一些最具挑战性问题的有前途的范例。然而，MAL系统的去中心化性质和代理的探索行为引入了独立机器学习系统和传统分布式系统中看不到的新漏洞。该项目旨在开发一种数据驱动的MAL安全方法，即使存在持续、协调和隐形的恶意内部人员或外部对手，也可以提供足够的保护。该项目的主要新奇之处在于，通过将对手建模和适应以原则性的方式纳入安全相关决策，超越了基于战略的攻击和防御计划。该项目通过整合网络安全，多智能体系统，机器学习和认知科学的跨学科方法，为安全科学和值得信赖的人工智能的新兴领域做出贡献。该项目的跨学科性质也为课程开发和学生培训带来了独特的机会。为大规模MAL系统开发强大的防御面临着由恶意代理的隐藏行为模式、环境的动态性和不确定性以及在许多隐私敏感设置中保护良性代理的本地数据的必要性引起的根本挑战。该项目通过在三个研究方向中逐步开发用于对抗性决策的（机器）心理理论来应对挑战。第一个推力是针对联邦和分散式机器学习系统开发基于学习的有针对性和无针对性的攻击。这些攻击首先从公开可用的数据中推断出世界模型，然后应用基于模型的强化学习来识别可以充分利用系统漏洞的自适应攻击策略。第二个推力研究了一个主动防御框架，该框架结合了对抗训练和本地适应，利用第一个推力中开发的自动攻击框架作为对手的模拟器，以获得强大的防御。第三个推力研究合作多智能体强化学习系统的安全性，通过解决一系列新的挑战，包括智能体之间的复杂交互，非平稳性和部分可观测性。其目标是了解恶意攻击和欺骗如何阻止良性代理达到社会偏好的结果，以及如何在完全合作和混合动机设置中解释更高的信念可以帮助代理（良性或恶意）。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Cooperative multi-agent learning (MAL), where multiple intelligent agents learn to coordinate with each other and with humans, is emerging as a promising paradigm for solving some of the most challenging problems in various security and safety-critical domains, including transportation, power systems, robotics, and healthcare. The decentralized nature of MAL systems and agents' exploration behavior, however, introduce new vulnerabilities unseen in standalone machine learning systems and traditional distributed systems. This project aims to develop a data-driven approach to MAL security that can provide an adequate level of protection even in the presence of persistent, coordinated, and stealthy malicious insiders or external adversaries. The main novelty of the project is to go beyond heuristics-based attack and defense schemes by incorporating opponent modeling and adaptation into security-related decision-making in a principled way. The project contributes to the emerging fields of science of security and trustworthy artificial intelligence via a cross-disciplinary approach that integrates cybersecurity, multi-agent systems, machine learning, and cognitive science. The interdisciplinary nature of this project also brings unique opportunities for both curriculum development and student training.Developing robust defenses for large-scale MAL systems faces fundamental challenges induced by the hidden behavioral patterns of malicious agents, the dynamics and uncertainty of the environment, and the necessity of protecting benign agents' local data in many privacy-sensitive settings. This project tackles the challenges by incrementally developing a (machine) theory of mind for adversarial decision-making in three research thrusts. The first thrust develops learning-based targeted and untargeted attacks against federated and decentralized machine learning systems. These attacks first infer a world model from publicly available data and then apply model-based reinforcement learning to identify an adaptive attack policy that can fully exploit the vulnerabilities of the systems. The second thrust investigates a proactive defense framework that combines adversarial training and local adaptation, utilizing the automated attack framework developed in the first thrust as a simulator of adversaries to obtain robust defenses. The third thrust studies security in cooperative multi-agent reinforcement learning systems by addressing a set of new challenges, including complicated interactions among agents, non-stationarity, and partial observability. The goal is to understand how malicious attacks and deceptions can prevent benign agents from reaching a socially preferred outcome and how accounting for a higher order of beliefs can help an agent (benign or malicious) in both fully cooperative and mixed-motive settings.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.