CISE-ANR: RI: Small: Numerically efficient reinforcement learning for constrained systems with super-linear convergence (NERL)

强化学习是一种用于教授机器人新技能的学习技术。强化学习也用于训练机器人以某些方式行为，这种训练可以实现从游戏到包裹处理的重要应用。然而，这些结果很少导致现实世界的大规模应用。主要挑战源于强化学习技术的有限能力，无法有效地创建可在现实环境中使用的行为，跨越物体，障碍和任务，同时仍然确保操作安全。最优控制是另一种用于控制系统的方法。这种方法可以非常有效地执行数值计算，但通常限于相当狭窄的行为。该项目旨在为强化学习和最优控制提供新的视角，这两种方法具有共同的基础，但到目前为止还未能产生一种结合两种方法优势的单一方法。这项研究将包括开发新的方法来提高学习效率和保证真实物理系统的安全性。为了证明该方法的广泛适用性，该项目将在四个实际应用领域评估这些方法：用于能源供应的拖风筝、有胳膊和腿的机器人、虚拟化身和蛋白质的微观运动。该项目通过提高机器人在具有高工业潜力的广泛应用领域的能力、可靠性和安全性，为促进国家健康、繁荣和福利作出贡献。该项目将由一个法国-美国研究团队进行，该团队将通过提供独特的国际研究经验来帮助培训下一代劳动力。该项目围绕两个主要研究目标进行阐述。第一个目标是产生一种新的强化学习算法，该算法更好地利用先验模型知识，特别是模型导数，以加速收敛，保证收敛速度并执行硬约束。第二个目标是有效地解决一类特殊的难题，即混合（离散/连续）动力学问题。这两个目标的结合将产生一个共同的理论框架，以合并最优控制和强化学习方法，以及能够普遍解决复杂高维问题的数值高效算法。最后，该项目的附带成果将是一些具有超越科学价值的示范。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Reinforcement learning is the name of a learning technique used to teach robots new skills. Reinforcement Learning is also used in training the robot behave in certain ways and this training enables important applications from game playing to package handling. Yet, these results seldom lead to real-world large-scale applications. The main challenges stem from the limited abilities of reinforcement learning techniques to efficiently create behaviors that can be used in realistic environments, across objects, obstacles and tasks, while still ensuring operational safety. Optimal control is another method that is used to control systems. Such methods can be very efficient for performing numerical computations but are generally limited to rather narrow behaviors. The project aims at casting a new light on reinforcement learning and optimal control, which share common foundations but until now have failed to produce a single method combining the advantages of both approaches. This research will include the development of new methods to improve learning efficacy and guarantee safety for real physical systems. To demonstrate the broad applicability of the approach, the project will evaluate the methods in four realistic application domains: towing kites for energy supply, robots with arms and legs, avatars and microscopic movement of proteins. This project contributes to the advance of national health, prosperity and welfare by improving the capabilities, reliability and safety of robots in a wide area of applications with high industrial potential. The project will be conducted by a French-US team of researchers which will help train the next generation of the workforce by providing a unique international research experience.The project is articulated around two main research goals. The first goal is to produce a new reinforcement learning algorithm which better exploits prior model knowledge, in particular model derivatives, to accelerate convergence, guarantee a convergence rate and enforce hard constraints. The second goal aims to efficiently solve a particular class of hard problems, namely problems with hybrid (discrete/continuous) dynamics. The combination of both objectives will result in a common theoretical framework to merge optimal control and reinforcement learning approaches as well as numerically efficient algorithms capable of generically solving complex high-dimensional problems. Finally, the side outcomes of the project will be several demonstrations which have value beyond the science.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.