Towards Robust and Safe Reinforcement Learning with Benign Off-policy Data
Towards Robust and Safe Reinforcement Learning with Benign Off-policy Data
复制标题
DOI：
--
复制
发表时间：
2023
期刊：
影响因子：
--
通讯作者：
Zuxin Liu;Zijian Guo;Zhepeng Cen;Huan Zhang;Yi-Fan Yao;Hanjiang Hu;Ding Zhao
中科院分区：
文献类型：
--
作者：
Zuxin Liu;Zijian Guo;Zhepeng Cen;Huan Zhang;Yi-Fan Yao;Hanjiang Hu;Ding Zhao
以前的工作表明，无噪声环境中的最佳安全强化学习策略是脆弱的，并且在观察攻击下可能是不安全的。虽然对抗训练有效地提高了鲁棒性和安全性，但在许多应用中，通过在线攻击行为代理来收集样本可能是昂贵的或非常危险的。我们提出了鲁棒的F-策略学习（SAFER）方法，它只需要良性的训练数据，而不攻击代理。SAFER通过凸优化获得最优的非参数变分策略分布，然后使用它通过监督学习鲁棒地改进参数化策略。两阶段的策略优化促进了强大的训练，在多个机器人平台上的广泛实验表明，SAFER在学习强大而安全的策略方面的效率：在训练过程中，以比策略基线少得多的约束违反实现相同的奖励。
Previous work demonstrates that the optimal safe reinforcement learning policy in a noise-free environment is vulnerable and could be un-safe under observational attacks. While adversarial training effectively improves robustness and safety, collecting samples by attacking the behavior agent online could be expensive or prohibitively dangerous in many applications. We propose the robuSt vAriational ofF-policy lEaRning (SAFER) approach, which only requires benign training data without at-tacking the agent. SAFER obtains an optimal non-parametric variational policy distribution via convex optimization and then uses it to improve the parameterized policy robustly via supervised learning. The two-stage policy optimization facilitates robust training, and extensive experiments on multiple robot platforms show the efficiency of SAFER in learning a robust and safe policy: achieving the same reward with much fewer constraint violations during training than on-policy baselines.