DOPE: Doubly Optimistic and Pessimistic Exploration for Safe Reinforcement Learning
DOPE: Doubly Optimistic and Pessimistic Exploration for Safe Reinforcement Learning
复制标题
DOI：
--
复制
发表时间：
2021-12
期刊：
影响因子：
--
通讯作者：
Archana Bura;Aria HasanzadeZonuzy;D. Kalathil;S. Shakkottai;J. Chamberland
中科院分区：
文献类型：
--
作者：
Archana Bura;Aria HasanzadeZonuzy;D. Kalathil;S. Shakkottai;J. Chamberland
安全的加固学习极具挑战性 - 不仅必须探索未知环境，还必须在确保没有安全约束的同时这样做。我们使用有限的马尔可夫决策过程（CMDP）制定了这种安全的加强学习（RL）问题，并具有未知的过渡概率函数，在此我们将安全要求作为对预期累积成本的约束对必须满足的预期累积成本进行建模。所有学习情节。我们提出了一种基于模型的安全RL算法，我们称之为双重乐观和悲观的探索（DOPE），并表明它实现了目标遗憾$ \ tilde {o}（| \ Mathcal {s}} | \ sqrt { a} |。 $ | \ MATHCAL {S} | $是状态数，$ | \ Mathcal {a} | $是动作数，而$ k $是学习情节的数量。我们的关键思想是除了基于标准的乐观模型探索外，还要将探索（乐观）与保守的约束（悲观）相结合。与早期的乐观情绪方法相比，涂料不仅能够改善客观遗憾的束缚，而且还显示出显着的经验绩效改善。
Safe reinforcement learning is extremely challenging--not only must the agent explore an unknown environment, it must do so while ensuring no safety constraint violations. We formulate this safe reinforcement learning (RL) problem using the framework of a finite-horizon Constrained Markov Decision Process (CMDP) with an unknown transition probability function, where we model the safety requirements as constraints on the expected cumulative costs that must be satisfied during all episodes of learning. We propose a model-based safe RL algorithm that we call Doubly Optimistic and Pessimistic Exploration (DOPE), and show that it achieves an objective regret $\tilde{O}(|\mathcal{S}|\sqrt{|\mathcal{A}| K})$ without violating the safety constraints during learning, where $|\mathcal{S}|$ is the number of states, $|\mathcal{A}|$ is the number of actions, and $K$ is the number of learning episodes. Our key idea is to combine a reward bonus for exploration (optimism) with a conservative constraint (pessimism), in addition to the standard optimistic model-based exploration. DOPE is not only able to improve the objective regret bound, but also shows a significant empirical performance improvement as compared to earlier optimism-pessimism approaches.