Near-optimal Conservative Exploration in Reinforcement Learning under Episode-wise Constraints
Near-optimal Conservative Exploration in Reinforcement Learning under Episode-wise Constraints
复制标题
DOI：
10.48550/arxiv.2306.06265
复制
发表时间：
2023-06
期刊：
影响因子：
2.4
通讯作者：
Donghao Li;Ruiquan Huang;Cong Shen;Jing Yang
中科院分区：
文献类型：
--
作者：
Donghao Li;Ruiquan Huang;Cong Shen;Jing Yang
本文研究了强化学习中的保守探索，其中学习代理的性能在整个学习过程中保证高于某个阈值。它重点关注具有有限状态和动作的表格情景马尔可夫决策过程 (MDP) 设置。根据现有安全基线策略的知识，提出了一种称为 StepMix 的算法来平衡利用和探索，同时确保在每个情节中都不会以高概率违反保守约束。 StepMix 采用独特的混合策略设计，可在基准策略和乐观策略之间自适应且平滑地进行插值。理论分析表明，StepMix 在无约束设置中实现了接近最优的后悔顺序，这表明遵守严格的情节保守约束不会损害学习性能。此外，还提出了基于随机化的 EpsMix 算法，并证明其性能与 StepMix 相同。将算法设计和理论分析进一步扩展到基线策略没有先验给出而必须从离线数据集中学习的情况，并证明如果离线数据集足够大，可以实现类似的保守保证和遗憾。实验结果证实了理论分析并证明了所提出的保守勘探策略的有效性。
This paper investigates conservative exploration in reinforcement learning where the performance of the learning agent is guaranteed to be above a certain threshold throughout the learning process. It focuses on the tabular episodic Markov Decision Process (MDP) setting that has finite states and actions. With the knowledge of an existing safe baseline policy, an algorithm termed as StepMix is proposed to balance the exploitation and exploration while ensuring that the conservative constraint is never violated in each episode with high probability. StepMix features a unique design of a mixture policy that adaptively and smoothly interpolates between the baseline policy and the optimistic policy. Theoretical analysis shows that StepMix achieves near-optimal regret order as in the constraint-free setting, indicating that obeying the stringent episode-wise conservative constraint does not compromise the learning performance. Besides, a randomization-based EpsMix algorithm is also proposed and shown to achieve the same performance as StepMix. The algorithm design and theoretical analysis are further extended to the setting where the baseline policy is not given a priori but must be learned from an offline dataset, and it is proved that similar conservative guarantee and regret can be achieved if the offline dataset is sufficiently large. Experiment results corroborate the theoretical analysis and demonstrate the effectiveness of the proposed conservative exploration strategies.