Curse-of-dimensionality-free nonlinear optimal feedback control with deep neural networks. A compositionality-based approach via Hamilton-Jacobi-Bellman PDEs

最优反馈控制是深度学习方法产生巨大影响的领域之一。深度强化学习是获得最优反馈规律的方法之一，可以说是人工智能领域最成功的算法之一，它是人工智能在国际象棋或围棋等游戏中惊人表现的背后，但在科学、技术和经济领域也有多种应用。从数学上讲，该方法背后的核心问题是如何通过深度神经网络（dnn）最好地表示最优值函数，即通过深度神经网络（dnn）为每个状态分配最佳性能值的函数，也称为强化学习中的成本函数。然后可以从这些函数中计算出最优反馈律。在连续时间中，这些最优值函数用Hamilton-Jacobi-Bellman偏微分方程（HJB PDEs）来表征，该方程通过dnn将问题与PDEs的解联系起来。由于HJB方程的维数是由控制最优控制问题的动力学状态维数决定的，因此HJB方程自然形成了一类高维PDE。因此，它们很容易受到众所周知的维度诅咒的影响，即其解的数值努力在维度上呈指数级增长。已知具有某些有益结构的函数，如组合函数或可分离函数，可以通过具有适当结构的深度神经网络来近似，避免了维数的诅咒。对于表征Lyapunov函数的HJB偏微分方程，该项目的提出者最近表明，小增益条件-即问题动力学的特定条件-建立了可分离子解的存在性，可以利用dnn通过具有适当损失函数的训练算法有效地逼近它们。这些结果为一般非线性HJB方程的无维数诅咒的基于dnn的方法铺平了道路，这是本项目的目标。除了小增益理论之外，还有大量的非线性反馈控制设计技术，这些技术可以导致组合（次）最优值函数。一方面，这些方法在数学上是合理的，适用于许多现实世界的问题，但另一方面，当计算结果的值函数或反馈律时，它们带来了重大的计算挑战。在本项目中，我们将利用这些方法提供的结构洞察力来建立组合最优值函数或其近似值的存在性，但通过使用适当的dnn训练算法来规避其计算复杂性。通过这种方式，我们将描述最优反馈控制问题，其中通过dnn的无维化（近似）解决方案是可能的，并提供有效的网络架构和训练方案来计算这些解决方案。

Optimal feedback control is one of the areas in which methods from deep learning have an enormous impact. Deep Reinforcement Learning, one of the methods for obtaining optimal feedback laws and arguably one of the most successful algorithms in artificial intelligence, stands behind the spectacular performance of artificial intelligence in games such as Chess or Go, but has also manifold applications in science, technology and economy. Mathematically, the core question behind this method is how to best represent optimal value functions, i.e., the functions that assign the optimal performance value to each state, also known as cost-to-go function in reinforcement learning, via deep neural networks (DNNs). The optimal feedback law can then be computed from these functions. In continuous time, these optimal value functions are characterised by Hamilton-Jacobi-Bellman partial differential equation (HJB PDEs), which links the question to the solution of PDEs via DNNs. As the dimension of the HJB PDE is determined by the dimension of the state of the dynamics governing the optimal control problem, HJB equations naturally form a class of high-dimensional PDEs. They are thus prone to the well-known curse of dimensionality, i.e., to the fact that the numerical effort for its solution grows exponentially in the dimension. It is known that functions with certain beneficial structures, like compositional or separable functions, can be approximated by DNNs with suitable architecture avoiding the curse of dimensionality. For HJB PDEs characterising Lyapunov functions it was recently shown by the proposer of this project that small-gain conditions - i.e., particular conditions on the dynamics of the problem - establish the existence of separable subsolutions, which can be exploited for efficiently approximating them by DNNs via training algorithms with suitable loss functions. These results pave the way for curse-of-dimensionality free DNN-based approaches for general nonlinear HJB equations, which are the goal of this project. Besides small-gain theory, there exists a large toolbox of nonlinear feedback control design techniques that lead to compositional (sub)optimal value functions. On the one hand, these methods are mathematically sound and apply to many real-world problems, but on the other hand they come with significant computational challenges when the resulting value functions or feedback laws shall be computed. In this project, we will exploit the structural insight provided these methods for establishing the existence of compositional optimal value functions or approximations thereof, but circumvent their computational complexity by using appropriate training algorithms for DNNs instead. Proceeding this way, we will characterise optimal feedback control problems for which curse-of-dimensionality-free (approximate) solutions via DNNs are possible and provide efficient network architectures and training schemes for computing these solutions.