The Generality and Practicality of Reinforcement Learning for Automatic Control

强化学习和动态规划之间理论关系的最新发现为开发自动控制器提供了令人兴奋的可能性，这些控制器通过经验学习来遵循最优控制策略。将强化学习算法与神经网络提供的自适应结构相结合，可以得到比现有自适应控制技术更灵活、更通用的理论上最优控制器。然而，要使强化学习网络实用，必须提高它们的学习效率。在以前的工作中，PI发现学习缓慢的一个原因是难以通过网络的隐藏单元发现有用的特征。在有监督学习范例中也认识到了这一困难，并证明了一些替代常见误差反向传播算法的方法可以显著减少学习时间。这些想法将被扩展到强化学习范例，并将探索它们在减少基于强化的网络的学习时间方面的潜力。其目的是缓解训练隐含单元的问题，并找出强化学习网络作为实时控制技术限制其通用性和实用性的任何剩余局限性。这些方法将包括模拟研究和作为物理系统控制器的实现。

Recent discoveries of the theoretical relationships between reinforcement learning and dynamic programming suggest exciting possibilities for developing automatic controllers that learn with experience to follow optimal control strategies. Combining reinforcement learning algorithms with the adaptive structure that neural networks provide results in theoretically optimal controllers that have more flexibility, and thus are more general, than current adaptive control techniques. However, for reinforcement learning networks to be practical, the efficiency with which they learn must be improved. In previous work, the PI identified one cause of slow learning to be difficulty of discovering useful features by the hidden units of the network. This difficulty has also been recognized within the supervisedlearning paradigm and a number of alternatives to the common error back propagation algorithm have been shown to significantly reduce learning time. These ideas will be extended to the reinforcement learning paradigm and their potential for reducing the learning time of reinforcement based networks will be explored. The objective is to alleviate the problem of training hidden units and to identify any remaining limitations of reinforcement learning networks that restrict their generality and practicality as real time control techniques. The methods will include both simulation studies and implementations as controllers of physical systems.