CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learning-猫眼课题宝

DOI：

10.48550/arxiv.2302.04782

发表时间：

2023-02

期刊：

ArXiv

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Sheng Yue;Guan Wang;Wei Shao;Zhaofeng Zhang;Sen Lin;Junkai Ren;Junshan Zhang

该工作旨在解决离线反向强化学习(IRL)中的一个主要挑战，即奖励外推误差，其中学习的奖励函数可能无法正确解释任务，并由于固有的协变量漂移而在不可见的环境中误导代理。利用专家数据和质量较低的多样性数据，我们设计了一种原则性算法(即CLARE)，通过将保守性整合到学习的奖励函数中并利用估计的动力学模型来有效地解决离线IRL问题。我们的理论分析提供了学习策略和专家策略之间回报差距的上界，在此基础上，我们通过检验开发(专家和多样化数据)和探索(估计的动态模型)之间的微妙双层权衡来表征协变量转移的影响。我们表明，Clare可以通过在其中找到正确的开发-勘探平衡来证明减轻报酬外推误差。广泛的实验证实了Clare在MuJoCo连续控制任务(特别是在小的离线数据集)上比现有最先进的算法有显著的性能提升，并且学习的回报对进一步的学习具有很高的指导意义。

This work aims to tackle a major challenge in offline Inverse Reinforcement Learning (IRL), namely the reward extrapolation error, where the learned reward function may fail to explain the task correctly and misguide the agent in unseen environments due to the intrinsic covariate shift. Leveraging both expert data and lower-quality diverse data, we devise a principled algorithm (namely CLARE) that solves offline IRL efficiently via integrating"conservatism"into a learned reward function and utilizing an estimated dynamics model. Our theoretical analysis provides an upper bound on the return gap between the learned policy and the expert policy, based on which we characterize the impact of covariate shift by examining subtle two-tier tradeoffs between the exploitation (on both expert and diverse data) and exploration (on the estimated dynamics model). We show that CLARE can provably alleviate the reward extrapolation error by striking the right exploitation-exploration balance therein. Extensive experiments corroborate the significant performance gains of CLARE over existing state-of-the-art algorithms on MuJoCo continuous control tasks (especially with a small offline dataset), and the learned reward is highly instructive for further learning.

CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learning

文献摘要

相似文献