Approximate Midpoint Policy Iteration for Linear Quadratic Control
Approximate Midpoint Policy Iteration for Linear Quadratic Control
复制标题
DOI：
--
复制
发表时间：
2020-11
期刊：
影响因子：
--
通讯作者：
Benjamin J. Gravell;I. Shames;T. Summers
中科院分区：
文献类型：
--
作者：
Benjamin J. Gravell;I. Shames;T. Summers
我们提出了一种中点策略迭代算法来解决基于模型和无模型设置中的线性二次最优控制问题。该算法是牛顿方法的变体，我们表明在基于模型的设置中它实现了三次收敛，这优于分别实现二次收敛和线性收敛的标准策略迭代和策略梯度算法。我们还证明，通过使用轨迹数据中的状态动作价值函数的最小二乘估计，可以在不了解动态模型的情况下近似实现该算法，从中可以获得策略改进。有了足够的轨迹数据，策略迭代三次收敛到近似最优策略，并且这在与近似标准策略迭代相同的可用样本预算的情况下发生。数值实验证明了所提出算法的有效性。
We present a midpoint policy iteration algorithm to solve linear quadratic optimal control problems in both model-based and model-free settings. The algorithm is a variation of Newton's method, and we show that in the model-based setting it achieves cubic convergence, which is superior to standard policy iteration and policy gradient algorithms that achieve quadratic and linear convergence, respectively. We also demonstrate that the algorithm can be approximately implemented without knowledge of the dynamics model by using least-squares estimates of the state-action value function from trajectory data, from which policy improvements can be obtained. With sufficient trajectory data, the policy iterates converge cubically to approximately optimal policies, and this occurs with the same available sample budget as the approximate standard policy iteration. Numerical experiments demonstrate effectiveness of the proposed algorithms.