Reinforcement Learning Control With Knowledge Shaping
Reinforcement Learning Control With Knowledge Shaping
复制标题
基于知识成形的强化学习控制
DOI：
10.1109/tnnls.2023.3243631
复制
发表时间：
2023-02
影响因子：
10.4
通讯作者：
Xiang Gao;Jennie Si;H. Huang
中科院分区：
文献类型：
--
作者：
Xiang Gao;Jennie Si;H. Huang
我们的目标是创建一个迁移强化学习框架，该框架允许学习控制器的设计利用从以前学习的任务和以前的数据中提取的先验知识来提高新任务的学习性能。为了实现这一目标，我们通过在我们的问题构造中的值函数中表达知识来形式化知识转移，这被称为具有知识塑造的强化学习（RL-KS）。与大多数迁移学习研究不同，我们的研究结果不仅包括模拟验证，还包括算法收敛性和解决方案最优性的分析。也不同于建立在政策不变性的证明基础上的基于潜力的奖励形成方法，我们的RL-KS方法使我们能够朝着一个新的理论结果前进，积极的知识转移。此外，我们的贡献包括两个原则性的方式，涵盖了一系列的实现方案，以表示在RL-KS先验知识。我们提供了广泛和系统的评价建议RL-KS方法。评估环境不仅包括经典的RL基准问题，而且还包括一个具有挑战性的任务，实时控制机器人下肢与人类用户在循环中。
We aim at creating a transfer reinforcement learning framework that allows the design of learning controllers to leverage prior knowledge extracted from previously learned tasks and previous data to improve the learning performance of new tasks. Toward this goal, we formalize knowledge transfer by expressing knowledge in the value function in our problem construct, which is referred to as reinforcement learning with knowledge shaping (RL-KS). Unlike most transfer learning studies that are empirical in nature, our results include not only simulation verifications but also an analysis of algorithm convergence and solution optimality. Also different from the well-established potential-based reward shaping methods which are built on proofs of policy invariance, our RL-KS approach allows us to advance toward a new theoretical result on positive knowledge transfer. Furthermore, our contributions include two principled ways that cover a range of realization schemes to represent prior knowledge in RL-KS. We provide extensive and systematic evaluations of the proposed RL-KS method. The evaluation environments not only include classical RL benchmark problems but also include a challenging task of real-time control of a robotic lower limb with a human user in the loop.