Lyapunov Methods for Reinforcement Learning

强化学习是随机最优控制问题逼近解的方法集合的总称。强化学习方法已经成功地应用于各种领域的大量此类问题，包括金融、物流、电信和机器人控制。尽管类似的问题在控制工程和运筹学领域已经深入研究了多年，但RL研究人员开发的方法为经典的求解方法增加了新的元素。RL方法提供了一种新颖的方法来近似解决那些太大或定义不清的问题，而经典的解决方法是可行的。强化学习研究的一个重要部分是通过为强化学习系统提供领域知识来提高在线性能和收敛速度。本课题涉及复杂动态系统稳定控制器的设计相关知识。它将尝试发展一种将这些知识纳入强化学习系统的一般理论。基本思想是用数学方法定义具有某些已知稳定性和安全性的策略子空间，并将重点放在这些策略子空间中的控制律上。实现这一目标的方法是基于李雅普诺夫稳定性理论和李雅普诺夫控制设计的相关方法

0070102BartoReinforcement learning is a summary term for a collection of methods for approximating solutions to stochastic optimal control problems. RL methods leave been successfully applied to a large array of such problems in a diversity of domains, including finance, logistics, telecommunications, and robot control. Although similar problems have been studied intensively for many years in control engineering and operations research, the methods developed by RL researchers have added new elements to the classical solution methods. RL methods offer novel ways to approximate solutions to problems that are too large or ill-defined for the classical solution methods to be feasible.A significant part of RL research is directed at increasing on-line performance and speed of convergence by providing RL systems with domain knowledge. This project is concerned with knowledge related to the design of stabilizing controllers for complex dynamical systems. It will try to develop a general theory for incorporating this knowledge into RL systems. The basic idea is to mathematically define policy subspaces that have certain known stability and safety properties and to focus exploration on control laws that lie within these policy subspaces. The means by which this is achieved are based on the theory of Lyapunov stability and the associated methods of Lyapunov control design.***