CAREER: Learning to Secure Cooperative Multi-Agent Learning Systems: Advanced Attacks and Robust Defenses

协作多代理学习（MAL）是指多个智能代理学习如何相互协调以及如何与人类协调，它正在成为解决各种安全和安全关键领域（包括交通、电力系统、机器人和医疗保健）中一些最具挑战性问题的一种有前途的范例。然而，MAL系统的分散性和代理的探索行为引入了在独立机器学习系统和传统分布式系统中看不到的新漏洞。该项目旨在开发一种数据驱动的MAL安全方法，即使在存在持久的、协调的和隐蔽的恶意内部人员或外部对手的情况下，也能提供足够的保护。该项目的主要新颖之处在于超越了基于启发式的攻击和防御方案，以一种原则性的方式将对手建模和适应整合到与安全相关的决策中。该项目通过集成网络安全、多智能体系统、机器学习和认知科学的跨学科方法，为安全和可信赖的人工智能科学的新兴领域做出贡献。该项目的跨学科性质也为课程开发和学生培训带来了独特的机会。为大规模MAL系统开发强大的防御面临着恶意代理的隐藏行为模式、环境的动态和不确定性以及在许多隐私敏感设置中保护良性代理的本地数据的必要性所带来的根本性挑战。该项目通过在三个研究重点中逐步开发对抗决策的（机器）心智理论来解决这些挑战。第一个重点是针对联邦和分散的机器学习系统开发基于学习的目标和非目标攻击。这些攻击首先从公开可用的数据中推断出一个世界模型，然后应用基于模型的强化学习来确定一个可以充分利用系统漏洞的自适应攻击策略。第二个推力研究了一个结合对抗训练和局部适应的主动防御框架，利用第一个推力中开发的自动攻击框架作为对手的模拟器来获得强大的防御。第三个重点是通过解决一系列新的挑战来研究合作多智能体强化学习系统的安全性，包括智能体之间复杂的相互作用、非平稳性和部分可观察性。我们的目标是理解恶意攻击和欺骗是如何阻止良性行为体达到社会偏好的结果的，以及在完全合作和混合动机的环境下，如何解释更高层次的信念可以帮助一个行为体（良性或恶意）。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Cooperative multi-agent learning (MAL), where multiple intelligent agents learn to coordinate with each other and with humans, is emerging as a promising paradigm for solving some of the most challenging problems in various security and safety-critical domains, including transportation, power systems, robotics, and healthcare. The decentralized nature of MAL systems and agents' exploration behavior, however, introduce new vulnerabilities unseen in standalone machine learning systems and traditional distributed systems. This project aims to develop a data-driven approach to MAL security that can provide an adequate level of protection even in the presence of persistent, coordinated, and stealthy malicious insiders or external adversaries. The main novelty of the project is to go beyond heuristics-based attack and defense schemes by incorporating opponent modeling and adaptation into security-related decision-making in a principled way. The project contributes to the emerging fields of science of security and trustworthy artificial intelligence via a cross-disciplinary approach that integrates cybersecurity, multi-agent systems, machine learning, and cognitive science. The interdisciplinary nature of this project also brings unique opportunities for both curriculum development and student training.Developing robust defenses for large-scale MAL systems faces fundamental challenges induced by the hidden behavioral patterns of malicious agents, the dynamics and uncertainty of the environment, and the necessity of protecting benign agents' local data in many privacy-sensitive settings. This project tackles the challenges by incrementally developing a (machine) theory of mind for adversarial decision-making in three research thrusts. The first thrust develops learning-based targeted and untargeted attacks against federated and decentralized machine learning systems. These attacks first infer a world model from publicly available data and then apply model-based reinforcement learning to identify an adaptive attack policy that can fully exploit the vulnerabilities of the systems. The second thrust investigates a proactive defense framework that combines adversarial training and local adaptation, utilizing the automated attack framework developed in the first thrust as a simulator of adversaries to obtain robust defenses. The third thrust studies security in cooperative multi-agent reinforcement learning systems by addressing a set of new challenges, including complicated interactions among agents, non-stationarity, and partial observability. The goal is to understand how malicious attacks and deceptions can prevent benign agents from reaching a socially preferred outcome and how accounting for a higher order of beliefs can help an agent (benign or malicious) in both fully cooperative and mixed-motive settings.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.