Algorithms for adaptive near-optimal control

点击翻译按钮获取中文摘要

In almost every field of science or engineering there are processes we would like to control in an optimal way, e.g. maximizing speed or accuracy or fuel efficiency. But optimal control is computationally so demanding that it is out of reach except for simple tasks. The next best thing may be near-optimal control, where we compute a sequence of better and better controllers, moving ever closer to the optimal one. There are many algorithms for this purpose, but the most efficient and versatile is probably the method of generalized Hamilton-Jacobi-Bellman (GHJB) equations. Here I show that this method is in an important sense indirect, and can be improved by using a more direct form of supervised learning. The GHJB method is based on the fact that if we have a feedback controller, and we learn to compute the gradient grad-J of its cost-to-go function, then we can use that gradient to define a better controller. We can then use the new controller's grad-J to define a still-better controller, and so on. But GHJB works indirectly in the sense that it doesn't learn the best approximation to grad-J but instead learns a related function and from that infers a suboptimal estimate of grad-J. I show how it is possible to learn the gradient directly; e.g. we need signals that report grad-J(x) for different states x of the controlled process, and I show how to obtain them using a formula similar to the Euler-Lagrange equation. I compare this direct algorithm with GHJB on test problems from recent control papers, and I show that the direct method yields controllers that are more nearly optimal and simpler, requiring (on one complex task) 10 times fewer function evaluations and adjustable parameters. But much more testing is needed, and there is a great deal of work to be done improving and extending this approach.