CIF: Small: Adversarially Robust Reinforcement Learning: Attack, Defense, and Analysis

为了开发可信的机器学习系统，有必要了解现有学习算法的潜在漏洞，然后制定相应的缓解策略。强化学习(RL)是一种控制理论问题的框架，可以在不确定的环境中随着时间的推移做出决策，在各种场景中有许多应用，例如推荐系统、自动驾驶、金融和企业管理等等。在现代行业规模的RL模型应用中，行动决策、奖励和状态信号收集以及策略迭代通常在分布式网络中实现。当包含奖励信号和行动决策的数据包通过网络传输时，攻击者可以拦截和修改这些数据包，以实施对抗性攻击。随着RL模型越来越多地被部署在安全关键和安全相关的应用中，迫切需要了解潜在的敌意攻击对这些应用的影响。在这个项目中，调查者旨在解决以下问题：1)RL代理做出的决策应该被信任吗？2)敌手可以误导RL代理吗？3)如何设计对对手攻击具有健壮性的RL算法？虽然许多已有的工作都是针对监督学习模型的对抗性攻击，但由于在线RL和监督学习之间的显著差异，人们对RL模型的脆弱性及其相应的缓解策略的理解还不够完整。特别是，与监督学习环境相比，RL模型的攻击/防御机制的设计和分析必须处理长期回报、无法访问未来数据和未知动态等挑战。该项目的目标是克服这些挑战，并初步尝试回答上述问题。这个项目的目标是：1)系统地调查RL模型和算法的潜在漏洞，2)开发可以减轻对抗性攻击影响的健壮RL算法，3)分析这些缓解策略的收益/成本。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

In order to develop trustworthy machine-learning systems, it is essential to understand the potential vulnerabilities of existing learning algorithms and then develop corresponding mitigation strategies. Reinforcement learning (RL), a framework for control-theoretic problems that makes decisions over time within uncertain environments, has many applications in a variety of scenarios, such as recommendation systems, autonomous driving, and finance and business management, to name a few. In modern industry-scale applications of RL models, action decisions, reward- and state-signal collection, and policy iterations are normally implemented in distributed networks. When data packets containing reward signals and action decisions are transmitted through the network, an attacker can intercept and modify these packets to implement adversarial attacks. As RL models are being increasingly deployed in safety-critical and security-related applications, there is a pressing need to understand the effects of potential adversarial attacks on these applications.In this project, the investigator aims to address the following questions: 1) Should decisions made by RL agents be trusted?; 2) Can an adversary mislead RL agents?; and 3) How to design RL algorithms that are robust to adversarial attacks? While many existing works address adversarial attacks on supervised learning models, the understandings of vulnerabilities of RL models and their corresponding mitigation strategies are less complete, partially due to the significant differences between online RL and supervised learning. In particular, compared with the supervised-learning setting, the design and analysis of attack/defense mechanisms for RL models have to handle challenges such as long-term rewards, no access to future data, and unknown dynamics. The goal of this project is to overcome these challenges and make initial attempts to answer the questions raised above. In particular, this project aims to: 1) systematically investigate potential vulnerabilities of RL models and algorithms, 2) develop robust RL algorithms that can mitigate the impacts of adversarial attacks, and 3) analyze the benefit/cost of these mitigation strategies.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.