CRII: RI: Secure Multi-Agent Reinforcement Learning Algorithms

近年来，强化学习（RL）取得了重大进展，这是机器学习的一个领域，在解决各种顺序决策问题方面取得了巨大成功。单智能体强化学习算法的进步引发了人们对多智能体强化学习（MARL）的新兴趣。该项目的目标是为使用MARL构建的自治系统构建健壮、安全的算法。项目团队将研究一种新的威胁，这种威胁可以简单地通过设计一个对抗计划来利用，让一个代理在合作的多代理环境中行动，从而创建一个或多个盟友对抗的自然观察。例如，在联网自动驾驶汽车中，一辆受损的车辆可能会彻底破坏安全性，造成混乱和错误，导致性能不佳，甚至伤害依赖这些系统的人类。项目团队将为这种对抗性操作构建一个健壮的MARL算法。这个项目的教育计划包括开发一套关于分析MARL算法的安全性和鲁棒性的教程，设计用于研究生课程或作为MARL研究人员的工具。项目团队还将通过让来自代表性不足群体的研究生和本科生参与进来，为教育推广做出贡献。该项目建立在三个总体目标之上：(1)研究攻击者如何利用MARL漏洞，(2)通过使用关于其他代理行为的反事实推理来训练每个代理来开发更健壮的MARL算法，以及(3)创建一种新颖的在线形式化验证方法，以满足我们提出的MARL算法执行期间的安全和安全要求。更具体地说，对于第一个目标，项目团队将证明使用受损代理通过其行动攻击其在MARL系统中的盟友的可行性。第二个目标将对攻击策略进行逆向工程，以开发一种鲁棒的MARL算法，该算法在训练期间模拟代理的行为，并使用反事实推理将其行为关联起来。在第三个目标中，将开发一个在线正式验证模型，使用一组预定义的安全和安全规范来检测代理行为中的任何偏差。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Recent years have witnessed significant advances in reinforcement learning (RL), an area of machine learning that achieved great success in solving various sequential decision-making problems. Advances in single‐agent RL algorithms sparked new interest in multi-agent RL (MARL). The goal of this project is to build robust, secure algorithms for autonomous systems that are built using MARL. The project team will investigate a novel threat that can be exploited simply by designing an adversarial plan for an agent acting in a cooperative multi‐agent environment so as to create natural observations that are adversarial to one or more of its allies. For example, in connected autonomous vehicles, one compromised vehicle candrastically disrupt security, causing confusion and mistakes that result in poor performance and even harm to humans who rely on these systems. The project team will build a robust MARL algorithm to such adversarial manipulations. The educational plan for this project includes developing a suit of tutorials on analyzing the security and robustness of MARL algorithms, designed for use in a graduate course or as a tool for MARL researchers. The project team will also contribute to educational outreach by involving graduate and undergraduate students from underrepresented groups.The project is built upon three overarching objectives (1) study how attackers can exploit MARL vulnerabilities,(2) develop a more robust MARL algorithm by training each agent using the counterfactual reasoning about other agents’ behaviors, and (3) create a novel online formal verification method to satisfy the security and safety requirements during the execution of our proposed MARL algorithm. More specifically, for the first objective, the project team will prove the feasibility of using a compromised agent to attack its allies in MARL systems through its actions. The second objective will reverse‐engineer the attack strategies to develop a robust MARL algorithm that models the agents’ behaviors during training and correlates their actions using counterfactual reasoning. In the third objective, an online formal verification model will be developed to detect any deviations in agents’ behaviors using a predefined set of security and safety specifications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.