Policy Poisoning in Batch Reinforcement Learning and Control
Policy Poisoning in Batch Reinforcement Learning and Control
复制标题
DOI：
--
复制
发表时间：
2019-10
期刊：
影响因子：
--
通讯作者：
Yuzhe Ma-;Xuezhou Zhang;Wen Sun;Xiaojin Zhu
中科院分区：
文献类型：
--
作者：
Yuzhe Ma-;Xuezhou Zhang;Wen Sun;Xiaojin Zhu
我们研究了批量强化学习和控制面临的一种安全威胁，在这种威胁中，攻击者旨在毒害所学策略。受害者是一个强化学习器/控制器，它首先从一个批量数据集中估计动态和奖励，然后根据这些估计求解最优策略。攻击者可以在学习发生之前稍微修改数据集，并希望迫使学习器学习攻击者选择的目标策略。我们提出了一个解决批量策略毒害攻击的统一框架，并在两个标准的受害者上实例化了攻击：强化学习中的表格确定性等价学习器和控制中的线性二次调节器。我们表明，这两种实例化都导致了一个凸优化问题，其全局最优性得到保证，并对攻击可行性和攻击成本进行了分析。实验表明了策略毒害攻击的有效性。
We study a security threat to batch reinforcement learning and control where the attacker aims to poison the learned policy. The victim is a reinforcement learner / controller which first estimates the dynamics and the rewards from a batch data set, and then solves for the optimal policy with respect to the estimates. The attacker can modify the data set slightly before learning happens, and wants to force the learner into learning a target policy chosen by the attacker. We present a unified framework for solving batch policy poisoning attacks, and instantiate the attack on two standard victims: tabular certainty equivalence learner in reinforcement learning and linear quadratic regulator in control. We show that both instantiation result in a convex optimization problem on which global optimality is guaranteed, and provide analysis on attack feasibility and attack cost. Experiments show the effectiveness of policy poisoning attacks.