Robust Deep Reinforcement Learning through Adversarial Loss
Robust Deep Reinforcement Learning through Adversarial Loss
复制标题
DOI：
--
复制
发表时间：
2020-08
期刊：
影响因子：
--
通讯作者：
Tuomas P. Oikarinen;Tsui-Wei Weng;L. Daniel
中科院分区：
文献类型：
--
作者：
Tuomas P. Oikarinen;Tsui-Wei Weng;L. Daniel
包括强化学习代理在内的深度神经网络已被证明容易受到输入中微小对抗性变化的影响，因此在真实的世界中部署此类网络存在问题。在本文中，我们提出了径向RL，一种方法来训练强化学习代理，提高对任何l_p$有界对抗攻击的鲁棒性。通过简单地最小化来自有效的鲁棒性验证方法的最坏情况下的对抗扰动下的损失函数的上限，我们显着提高了在Atari-2600游戏上训练的RL代理的鲁棒性，并表明当针对PGD攻击进行评估时，RADIAL-RL可以击败最先进的鲁棒训练算法。我们还提出了一种新的评估方法，贪婪最坏情况下的奖励（GWC），用于测量攻击不可知的RL代理的鲁棒性。GWC可以被有效地评估，并且它可以很好地估计最坏的对抗性攻击序列下的回报;特别是，GWC考虑了每个动作的重要性及其时间依赖性，改进了以前的方法，这些方法只评估每个动作是否可以在输入扰动下改变。我们的代码可以在这个https URL上找到。
Deep neural networks, including reinforcement learning agents, have been proven vulnerable to small adversarial changes in the input, thus making deploying such networks in the real world problematic. In this paper, we propose RADIAL-RL, a method to train reinforcement learning agents with improved robustness against any $l_p$-bounded adversarial attack. By simply minimizing an upper bound of the loss functions under worst case adversarial perturbation derived from efficient robustness verification methods, we significantly improve robustness of RL-agents trained on Atari-2600 games and show that RADIAL-RL can beat state-of-the-art robust training algorithms when evaluated against PGD-attacks. We also propose a new evaluation method, Greedy Worst-Case Reward (GWC), for measuring attack agnostic robustness of RL agents. GWC can be evaluated efficiently and it serves as a good estimate of the reward under the worst possible sequence of adversarial attacks; in particular, GWC accounts for the importance of each action and their temporal dependency, improving upon previous approaches that only evaluate whether each single action can change under input perturbations. Our code is available at this https URL.