Offline Reinforcement Learning with Closed-Form Policy Improvement Operators-猫眼课题宝

DOI：

10.48550/arxiv.2211.15956

发表时间：

2022-11

期刊：

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Jiachen Li;Edwin Zhang;Ming Yin;Qinxun Bai;Yu-Xiang Wang;William Yang Wang

行为约束策略优化已被证明是解决离线强化学习的成功范例。通过利用历史转换，策略被训练为最大化学习的值函数，同时受到行为策略的约束，以避免显著的分布变化。在本文中，我们提出了我们的封闭形式的政策改进运营商。我们做了一个新的观察，行为约束自然激励使用一阶泰勒近似，导致政策目标的线性近似。此外，由于实际数据集通常由异构策略收集，我们将行为策略建模为高斯混合模型，并利用LogSumExp的下界和詹森不等式克服了由此引起的优化困难，从而产生了一个封闭形式的策略改进算子。我们实例化离线RL算法与我们的新的政策改进运营商和经验证明其有效性超过国家的最先进的算法在标准的D4RL基准。我们的代码可在https://cfpi-icml23.github.io/上获得。

Behavior constrained policy optimization has been demonstrated to be a successful paradigm for tackling Offline Reinforcement Learning. By exploiting historical transitions, a policy is trained to maximize a learned value function while constrained by the behavior policy to avoid a significant distributional shift. In this paper, we propose our closed-form policy improvement operators. We make a novel observation that the behavior constraint naturally motivates the use of first-order Taylor approximation, leading to a linear approximation of the policy objective. Additionally, as practical datasets are usually collected by heterogeneous policies, we model the behavior policies as a Gaussian Mixture and overcome the induced optimization difficulties by leveraging the LogSumExp's lower bound and Jensen's Inequality, giving rise to a closed-form policy improvement operator. We instantiate offline RL algorithms with our novel policy improvement operators and empirically demonstrate their effectiveness over state-of-the-art algorithms on the standard D4RL benchmark. Our code is available at https://cfpi-icml23.github.io/.

Offline Reinforcement Learning with Closed-Form Policy Improvement Operators

文献摘要

相似文献