CIF: Small: Adversarially Robust Reinforcement Learning: Attack, Defense, and Analysis

为了开发值得信赖的机器学习系统，必须了解现有学习算法的潜在漏洞，然后制定相应的缓解策略。强化学习（RL）是一个控制理论问题的框架，它可以在不确定的环境中随着时间的推移做出决策，在各种场景中都有许多应用，例如推荐系统、自动驾驶、金融和商业管理等。在RL模型的现代工业规模应用中，行动决策、奖励和状态信号收集以及策略迭代通常在分布式网络中实现。当包含奖励信号和行动决定的数据包通过网络传输时，攻击者可以拦截和修改这些数据包以实现对抗性攻击。随着RL模型越来越多地部署在安全关键和安全相关的应用程序中，迫切需要了解潜在的对抗性攻击对这些应用程序的影响。在这个项目中，研究者旨在解决以下问题：1)RL代理做出的决定是否值得信任？2)对手可以误导RL特工吗？3)如何设计对对抗性攻击具有鲁棒性的强化学习算法？虽然许多现有的工作解决了对监督学习模型的对抗性攻击，但对强化学习模型的漏洞及其相应的缓解策略的理解并不完整，部分原因是在线强化学习和监督学习之间存在显著差异。特别是，与监督学习设置相比，强化学习模型的攻击/防御机制的设计和分析必须处理诸如长期奖励，无法访问未来数据和未知动态等挑战。这个项目的目标是克服这些挑战，并初步尝试回答上面提出的问题。特别是，该项目旨在：1)系统地调查RL模型和算法的潜在漏洞，2)开发可以减轻对抗性攻击影响的健壮的RL算法，以及3)分析这些缓解策略的收益/成本。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

In order to develop trustworthy machine-learning systems, it is essential to understand the potential vulnerabilities of existing learning algorithms and then develop corresponding mitigation strategies. Reinforcement learning (RL), a framework for control-theoretic problems that makes decisions over time within uncertain environments, has many applications in a variety of scenarios, such as recommendation systems, autonomous driving, and finance and business management, to name a few. In modern industry-scale applications of RL models, action decisions, reward- and state-signal collection, and policy iterations are normally implemented in distributed networks. When data packets containing reward signals and action decisions are transmitted through the network, an attacker can intercept and modify these packets to implement adversarial attacks. As RL models are being increasingly deployed in safety-critical and security-related applications, there is a pressing need to understand the effects of potential adversarial attacks on these applications.In this project, the investigator aims to address the following questions: 1) Should decisions made by RL agents be trusted?; 2) Can an adversary mislead RL agents?; and 3) How to design RL algorithms that are robust to adversarial attacks? While many existing works address adversarial attacks on supervised learning models, the understandings of vulnerabilities of RL models and their corresponding mitigation strategies are less complete, partially due to the significant differences between online RL and supervised learning. In particular, compared with the supervised-learning setting, the design and analysis of attack/defense mechanisms for RL models have to handle challenges such as long-term rewards, no access to future data, and unknown dynamics. The goal of this project is to overcome these challenges and make initial attempts to answer the questions raised above. In particular, this project aims to: 1) systematically investigate potential vulnerabilities of RL models and algorithms, 2) develop robust RL algorithms that can mitigate the impacts of adversarial attacks, and 3) analyze the benefit/cost of these mitigation strategies.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.