CISE-ANR: RI: Small: Numerically efficient reinforcement learning for constrained systems with super-linear convergence (NERL)

强化学习是一种用于向机器人传授新技能的学习技术。强化学习也被用来训练机器人以某种方式行为，这种训练使从游戏到包裹处理的重要应用成为可能。然而，这些结果很少会导致真正的大规模应用程序。主要的挑战来自于强化学习技术在有效地创建可在现实环境中使用的行为的有限能力，这些行为可以跨越对象、障碍和任务，同时仍然确保操作安全。最优控制是用于控制系统的另一种方法。这种方法可以非常有效地执行数值计算，但通常限于相当狭窄的行为。该项目旨在为强化学习和最优控制提供新的视角，这两种方法有着共同的基础，但到目前为止还没有产生一种结合这两种方法的优点的单一方法。这项研究将包括开发新的方法来提高学习效率并保证真实物理系统的安全。为了展示这种方法的广泛适用性，该项目将在四个现实应用领域对这些方法进行评估：用于能源供应的拖曳风筝、有胳膊和腿的机器人、化身和蛋白质的微观运动。该项目通过提高机器人在具有高工业潜力的广泛应用领域的能力、可靠性和安全性，为促进国家健康、繁荣和福利做出贡献。该项目将由一个由法国和美国组成的研究团队进行，通过提供独特的国际研究体验，帮助培训下一代劳动力。该项目围绕两个主要研究目标展开。第一个目标是产生一种新的强化学习算法，它能更好地利用先验模型知识，特别是模型导数，以加快收敛速度，保证收敛速度，并实施硬约束。第二个目标是有效地解决一类特殊的困难问题，即具有混合(离散/连续)动力学的问题。这两个目标的结合将导致一个共同的理论框架，以合并最优控制和强化学习方法，以及能够普遍解决复杂高维问题的数值高效算法。最后，该项目的附带成果将是几个具有超越科学价值的示范。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Reinforcement learning is the name of a learning technique used to teach robots new skills. Reinforcement Learning is also used in training the robot behave in certain ways and this training enables important applications from game playing to package handling. Yet, these results seldom lead to real-world large-scale applications. The main challenges stem from the limited abilities of reinforcement learning techniques to efficiently create behaviors that can be used in realistic environments, across objects, obstacles and tasks, while still ensuring operational safety. Optimal control is another method that is used to control systems. Such methods can be very efficient for performing numerical computations but are generally limited to rather narrow behaviors. The project aims at casting a new light on reinforcement learning and optimal control, which share common foundations but until now have failed to produce a single method combining the advantages of both approaches. This research will include the development of new methods to improve learning efficacy and guarantee safety for real physical systems. To demonstrate the broad applicability of the approach, the project will evaluate the methods in four realistic application domains: towing kites for energy supply, robots with arms and legs, avatars and microscopic movement of proteins. This project contributes to the advance of national health, prosperity and welfare by improving the capabilities, reliability and safety of robots in a wide area of applications with high industrial potential. The project will be conducted by a French-US team of researchers which will help train the next generation of the workforce by providing a unique international research experience.The project is articulated around two main research goals. The first goal is to produce a new reinforcement learning algorithm which better exploits prior model knowledge, in particular model derivatives, to accelerate convergence, guarantee a convergence rate and enforce hard constraints. The second goal aims to efficiently solve a particular class of hard problems, namely problems with hybrid (discrete/continuous) dynamics. The combination of both objectives will result in a common theoretical framework to merge optimal control and reinforcement learning approaches as well as numerically efficient algorithms capable of generically solving complex high-dimensional problems. Finally, the side outcomes of the project will be several demonstrations which have value beyond the science.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.