Corruption-Robust Offline Reinforcement Learning
Corruption-Robust Offline Reinforcement Learning
复制标题
DOI：
--
复制
发表时间：
2021-06
期刊：
影响因子：
--
通讯作者：
Xuezhou Zhang;Yiding Chen;Jerry Zhu;Wen Sun
中科院分区：
文献类型：
--
作者：
Xuezhou Zhang;Yiding Chen;Jerry Zhu;Wen Sun
我们研究了离线强化学习的对抗鲁棒性。给定由元组$（s，a，r，s '）$组成的批处理数据集，允许攻击者任意修改元组的$\n $分数。从损坏的数据集，学习器的目标是鲁棒地识别一个接近最优的策略。我们首先表明，最坏情况下的$\Omega（d\）$最优性差距是不可避免的，在线性MDP的维度$d$，即使对手只损坏的奖励元素的元组。这与鲁棒监督学习中的无量纲结果和腐败在线RL设置中最知名的下限形成对比。接下来，我们提出了鲁棒的变体的最小二乘值迭代（LSVI）算法，利用强大的监督学习预言机，实现近匹配性能的情况下，有和没有完整的数据覆盖。该算法需要的知识$\n $设计的悲观奖金在无覆盖的情况下。令人惊讶的是，在这种情况下，知识的$\n $是必要的，因为我们表明，适应未知的$\n $是不可能的。这再次对比最近的结果腐败鲁棒的在线RL，并意味着强大的离线RL是一个严格困难的问题。
We study the adversarial robustness in offline reinforcement learning. Given a batch dataset consisting of tuples $(s, a, r, s')$, an adversary is allowed to arbitrarily modify $\epsilon$ fraction of the tuples. From the corrupted dataset the learner aims to robustly identify a near-optimal policy. We first show that a worst-case $\Omega(d\epsilon)$ optimality gap is unavoidable in linear MDP of dimension $d$, even if the adversary only corrupts the reward element in a tuple. This contrasts with dimension-free results in robust supervised learning and best-known lower-bound in the online RL setting with corruption. Next, we propose robust variants of the Least-Square Value Iteration (LSVI) algorithm utilizing robust supervised learning oracles, which achieve near-matching performances in cases both with and without full data coverage. The algorithm requires the knowledge of $\epsilon$ to design the pessimism bonus in the no-coverage case. Surprisingly, in this case, the knowledge of $\epsilon$ is necessary, as we show that being adaptive to unknown $\epsilon$ is impossible.This again contrasts with recent results on corruption-robust online RL and implies that robust offline RL is a strictly harder problem.