CRII: RI: Secure Multi-Agent Reinforcement Learning Algorithms

近年来，强化学习（RL）取得了重大进展，这是机器学习的一个领域，在解决各种顺序决策问题方面取得了巨大成功。单智能体RL算法的进步引发了对多智能体RL（MARL）的新兴趣。该项目的目标是为使用MARL构建的自治系统构建强大，安全的算法。项目团队将调查一种新的威胁，只需为在合作多智能体环境中行动的智能体设计一个对抗计划，就可以利用这种威胁，从而创建对其一个或多个盟友具有对抗性的自然观察。例如，在联网的自动驾驶汽车中，一辆受损的汽车可能会严重破坏安全性，导致混乱和错误，导致性能低下，甚至伤害依赖这些系统的人。项目团队将构建一个强大的MARL算法来应对这种对抗性操作。该项目的教育计划包括开发一套关于分析MARL算法的安全性和鲁棒性的教程，旨在用于研究生课程或作为MARL研究人员的工具。项目团队还将通过让来自代表性不足群体的研究生和本科生参与教育推广活动。该项目建立在三个总体目标之上（1）研究攻击者如何利用MARL漏洞，（2）通过使用关于其他代理行为的反事实推理来训练每个代理，（3）提出了一种新的在线形式化验证方法，以满足MARL算法在执行过程中的安全性要求。更具体地说，对于第一个目标，项目团队将通过其行动证明使用受损代理攻击其在MARL系统中的盟友的可行性。第二个目标将对攻击策略进行逆向工程，以开发一种强大的MARL算法，该算法在训练期间对代理的行为进行建模，并使用反事实推理将其行为关联起来。在第三个目标中，将开发一个在线正式验证模型，以使用一套预定义的安全和安全规范来检测代理行为中的任何偏差。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Recent years have witnessed significant advances in reinforcement learning (RL), an area of machine learning that achieved great success in solving various sequential decision-making problems. Advances in single‐agent RL algorithms sparked new interest in multi-agent RL (MARL). The goal of this project is to build robust, secure algorithms for autonomous systems that are built using MARL. The project team will investigate a novel threat that can be exploited simply by designing an adversarial plan for an agent acting in a cooperative multi‐agent environment so as to create natural observations that are adversarial to one or more of its allies. For example, in connected autonomous vehicles, one compromised vehicle candrastically disrupt security, causing confusion and mistakes that result in poor performance and even harm to humans who rely on these systems. The project team will build a robust MARL algorithm to such adversarial manipulations. The educational plan for this project includes developing a suit of tutorials on analyzing the security and robustness of MARL algorithms, designed for use in a graduate course or as a tool for MARL researchers. The project team will also contribute to educational outreach by involving graduate and undergraduate students from underrepresented groups.The project is built upon three overarching objectives (1) study how attackers can exploit MARL vulnerabilities,(2) develop a more robust MARL algorithm by training each agent using the counterfactual reasoning about other agents’ behaviors, and (3) create a novel online formal verification method to satisfy the security and safety requirements during the execution of our proposed MARL algorithm. More specifically, for the first objective, the project team will prove the feasibility of using a compromised agent to attack its allies in MARL systems through its actions. The second objective will reverse‐engineer the attack strategies to develop a robust MARL algorithm that models the agents’ behaviors during training and correlates their actions using counterfactual reasoning. In the third objective, an online formal verification model will be developed to detect any deviations in agents’ behaviors using a predefined set of security and safety specifications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.