Safe Exploration Incurs Nearly No Additional Sample Complexity for Reward-free RL
Safe Exploration Incurs Nearly No Additional Sample Complexity for Reward-free RL
复制标题
DOI：
10.48550/arxiv.2206.14057
复制
发表时间：
2022-06
期刊：
影响因子：
--
通讯作者：
Ruiquan Huang;J. Yang;Yingbin Liang
中科院分区：
文献类型：
--
作者：
Ruiquan Huang;J. Yang;Yingbin Liang
无奖励强化学习（RF-RL）是最近推出的一种强化学习范式，它依靠随机采取行动来探索未知环境，而无需任何奖励反馈信息。虽然 RF-RL 探索阶段的主要目标是用最少的轨迹数来减少估计模型的不确定性，但在实践中，智能体通常需要同时遵守一定的安全约束。目前尚不清楚这种安全探索要求将如何影响相应的样本复杂性，以实现规划中所获得策略的理想最优性。在这项工作中，我们首次尝试回答这个问题。特别是，我们考虑了事先已知安全基线策略的情况，并提出了一个统一的安全无奖励探索（SWEET）框架。然后，我们将 SWEET 框架具体化为表格和低秩 MDP 设置，并分别开发称为 Tabular-SWEET 和 Low-rank-SWEET 的算法。两种算法都利用了新引入的截断值函数的凹性和连续性，并保证在探索过程中以高概率实现零约束违规。此外，这两种算法都可以证明在规划阶段不受任何约束的情况下找到接近最优的策略。值得注意的是，两种算法下的样本复杂性在某些常数因子范围内都与无约束算法中的最新技术相匹配，甚至优于现有技术，证明安全约束几乎不会增加 RF-RL 的样本复杂性。
Reward-free reinforcement learning (RF-RL), a recently introduced RL paradigm, relies on random action-taking to explore the unknown environment without any reward feedback information. While the primary goal of the exploration phase in RF-RL is to reduce the uncertainty in the estimated model with minimum number of trajectories, in practice, the agent often needs to abide by certain safety constraint at the same time. It remains unclear how such safe exploration requirement would affect the corresponding sample complexity in order to achieve the desired optimality of the obtained policy in planning. In this work, we make a first attempt to answer this question. In particular, we consider the scenario where a safe baseline policy is known beforehand, and propose a unified Safe reWard-frEe ExploraTion (SWEET) framework. We then particularize the SWEET framework to the tabular and the low-rank MDP settings, and develop algorithms coined Tabular-SWEET and Low-rank-SWEET, respectively. Both algorithms leverage the concavity and continuity of the newly introduced truncated value functions, and are guaranteed to achieve zero constraint violation during exploration with high probability. Furthermore, both algorithms can provably find a near-optimal policy subject to any constraint in the planning phase. Remarkably, the sample complexities under both algorithms match or even outperform the state of the art in their constraint-free counterparts up to some constant factors, proving that safety constraint hardly increases the sample complexity for RF-RL.