NSF-AoF: RI: Small: Safe Reinforcement Learning in Non-Stationary Environments With Fast Adaptation and Disturbance Prediction

强化学习(RL)在控制复杂的机器人系统执行各种任务方面表现出了令人印象深刻的性能，如移动、操纵和进行运动，如乒乓球。强化学习使机器人能够通过与环境的反复试验相互作用，自主发现最优行为。然而，环境扰动很容易导致在旧环境中训练的行为策略在扰动环境中失败。对于自动驾驶汽车、无人机、飞行出租车和建筑机械等安全关键型机器人系统来说，这种故障是不可接受的。现有的稳健方法试图在训练阶段考虑所有场景，并寻求固定的策略，从而导致保守行为。现有的自适应方法试图在受干扰的环境中更新它们的行为策略，但只有在机器人通过与环境的相互作用感觉到不同之后才会这样做。相比之下，人类可以利用他/她的感知在新的环境中进行预测，并在与之互动之前相应地调整他/她的行为。根据这些条件，本项目设想了一个新的框架，用于在存在环境变化的情况下利用快速适应和基于感知的预测来安全和有效地进行RL。该框架将使机器人和自主系统能够稳健而安全地在现实世界中运行、学习和适应。该项目依赖于以下推力：i)用于安全和高效的策略更新的混合RL；ii)具有安全保证的稳健自适应控制；iii)基于视觉的干扰预测。更具体地说，该项目将开发稳健的自适应控制算法，以确保在环境变化引起的干扰存在时，机器人执行的轨迹保持安全。它将促进无模型/基于模型的混合RL算法，这些算法能够在控制算法的帮助下高效而安全地更新行为策略。该项目将提出新的方法，用于直接从图像观测中预测干扰的关键参数(例如，包的重量)，从而产生新的可扩展的方法，用于有效地学习具有量化误差范围的干扰的数学模型。所有的组成部分将被整体整合，以建立一个框架，使机器人能够安全、稳健、高效地在现实世界环境中操作和适应。空中和地面车辆将用于实验验证。这一奖励反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Reinforcement learning (RL) has shown impressive performance in the control of complex robotic systems for various tasks such as locomotion, manipulation, and playing sports, e.g., table tennis. Reinforcement learning enables a robot to autonomously discover an optimal behavior through trial-and-error interactions with its environment. However, the environmental perturbations could easily cause a behavior policy trained in an old environment to fail in a perturbed environment. The failure is unacceptable for safety-critical robotic systems such as self-driving cars, drones, flying taxies and construction machines. Existing robust methods try to consider all scenarios during the training phase and seek a fixed policy, leading to conservative behaviors. Existing adaptive methods try to update their behavior policies in the perturbed environment, but will only do that after the robot has “felt a difference” through its interaction with the environment. In contrast, a human could leverage his/her perception for prediction in the new environment and adjust his/her behavior accordingly even before interacting with it. In light of these conditions, this project envisions a new framework for safe and efficient RL in the presence of environmental changes leveraging fast adaptation and perception-based prediction. The framework will enable robotic and autonomous systems robustly and safely operate, learn and adapt in the real world. This project relies on the following thrusts: i) hybrid RL for safe and efficient policy updates, ii) robust adaptive control with safety guarantees; iii) vision-based disturbance prediction. More specifically, the project will develop robust adaptive control algorithms that ensure that the executed trajectory of a robot remains safe in the presence of disturbances induced by environmental changes. It will spur hybrid model-free/model-based RL algorithms that are capable of efficiently and safely updating the behavior policies with the help of the control algorithms. The project will advance novel methodologies for predicting the key parameters of the disturbances (e.g., the weight of a package) directly from the image observations, leading to new scalable methods for efficiently learning the mathematical model of the disturbances with quantified error bounds. All the ingredients will be holistically integrated to build a framework to enable robots to safely, robustly, and efficiently operate and adapt in real-world environments. Aerial and ground vehicles will be used for experimental validation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.