Algorithms for adaptive near-optimal control

在几乎每一个科学或工程领域，都有我们希望以最佳方式控制的过程，例如最大限度地提高速度或精度或燃油效率。但是，最优控制在计算上要求如此之高，以至于除了简单的任务之外，它是遥不可及的。下一个最好的事情可能是近最优控制，我们计算一系列越来越好的控制器，越来越接近最优控制器。有许多算法用于此目的，但最有效和通用的可能是广义Hamilton-Jacobi-Bellman（GHJB）方程的方法。在这里，我表明，这种方法在一个重要的意义上是间接的，可以通过使用更直接的监督学习形式来改进。GHJB方法基于这样一个事实：如果我们有一个反馈控制器，并且我们学会计算其成本函数的梯度grad-J，那么我们可以使用该梯度来定义更好的控制器。然后，我们可以使用新的控制器的grad-J来定义一个更好的控制器，等等。但是GHJB的工作是间接的，因为它不学习grad-J的最佳近似，而是学习一个相关的函数，并从中推断出grad-J的次优估计。例如，我们需要报告受控过程的不同状态x的grad-J（x）的信号，我展示了如何使用类似于欧拉-拉格朗日方程的公式获得它们。我比较这种直接算法与GHJB测试问题，从最近的控制论文，我表明，直接方法产生的控制器，更接近最佳和更简单，需要（在一个复杂的任务）10倍少的功能评估和可调参数。但还需要更多的测试，并且还有大量的工作要做来改进和扩展这种方法。

In almost every field of science or engineering there are processes we would like to control in an optimal way, e.g. maximizing speed or accuracy or fuel efficiency. But optimal control is computationally so demanding that it is out of reach except for simple tasks. The next best thing may be near-optimal control, where we compute a sequence of better and better controllers, moving ever closer to the optimal one. There are many algorithms for this purpose, but the most efficient and versatile is probably the method of generalized Hamilton-Jacobi-Bellman (GHJB) equations. Here I show that this method is in an important sense indirect, and can be improved by using a more direct form of supervised learning. The GHJB method is based on the fact that if we have a feedback controller, and we learn to compute the gradient grad-J of its cost-to-go function, then we can use that gradient to define a better controller. We can then use the new controller's grad-J to define a still-better controller, and so on. But GHJB works indirectly in the sense that it doesn't learn the best approximation to grad-J but instead learns a related function and from that infers a suboptimal estimate of grad-J. I show how it is possible to learn the gradient directly; e.g. we need signals that report grad-J(x) for different states x of the controlled process, and I show how to obtain them using a formula similar to the Euler-Lagrange equation. I compare this direct algorithm with GHJB on test problems from recent control papers, and I show that the direct method yields controllers that are more nearly optimal and simpler, requiring (on one complex task) 10 times fewer function evaluations and adjustable parameters. But much more testing is needed, and there is a great deal of work to be done improving and extending this approach.