Hamiltonian-Driven Adaptive Dynamic Programming With Approximation Errors
Hamiltonian-Driven Adaptive Dynamic Programming With Approximation Errors
复制标题
具有近似误差的哈密顿驱动自适应动态规划
DOI：
10.1109/tcyb.2021.3108034
复制
发表时间：
2022
影响因子：
11.8
通讯作者：
Donald C. Wunsch
中科院分区：
文献类型：
--
作者：
Yongliang Yang;Hamidreza Modares;Kyriakos G. Vamvoudakis;Wei He;Cheng-Zhong Xu;Donald C. Wunsch
在这篇文章中，我们考虑了哈密顿驱动框架下的迭代自适应动态规划(ADP)算法来求解连续时间非线性系统无限范围最优控制问题的哈密顿-雅可比-贝尔曼(HJB)方程。首先，定义了一种新的函数“最小哈密顿量”来捕捉经典哈密顿量的基本性质。结果表明，在哈密顿驱动的框架下，HJB方程和策略迭代(PI)算法都可以表示为最小哈密顿量。此外，我们还提出了一种迭代ADP算法，该算法考虑了策略评估过程中的逼近误差。在此基础上，给出了保证平衡点的闭环稳定性和收敛于最优值的迭代值梯度的充分条件。还提供了一种基于非策略强化学习(RL)技术的无模型扩展。最后，数值结果验证了该框架的有效性。
In this article, we consider an iterative adaptive dynamic programming (ADP) algorithm within the Hamiltonian-driven framework to solve the Hamilton–Jacobi–Bellman (HJB) equation for the infinite-horizon optimal control problem in continuous time for nonlinear systems. First, a novel function, “min-Hamiltonian,” is defined to capture the fundamental properties of the classical Hamiltonian. It is shown that both the HJB equation and the policy iteration (PI) algorithm can be formulated in terms of the min-Hamiltonian within the Hamiltonian-driven framework. Moreover, we develop an iterative ADP algorithm that takes into consideration the approximation errors during the policy evaluation step. We then derive a sufficient condition on the iterative value gradient to guarantee closed-loop stability of the equilibrium point as well as convergence to the optimal value. A model-free extension based on an off-policy reinforcement learning (RL) technique is also provided. Finally, numerical results illustrate the efficacy of the proposed framework.