"Good Robot! Now Watch This!": Repurposing Reinforcement Learning for Task-to-Task Transfer
"Good Robot! Now Watch This!": Repurposing Reinforcement Learning for Task-to-Task Transfer
复制标题
DOI：
--
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Andrew Hundt;Aditya Murali;Priyanka Hubli;Ran Liu;N. Gopalan;M. Gombolay;Gregory Hager
中科院分区：
文献类型：
--
作者：
Andrew Hundt;Aditya Murali;Priyanka Hubli;Ran Liu;N. Gopalan;M. Gombolay;Gregory Hager
现代强化学习（RL）算法在复杂领域的多步任务上训练时样本效率不高，阻碍了它们在真实的世界中的更广泛部署。我们通过利用这样的洞察力来解决这个问题，即经过训练以完成一组任务的RL模型可以在仅给出少量演示时重新用于完成相关任务。基于这一见解，我们提出了See-SPOT-Run（SSR），这是一种新的机器人学习计算方法，使机器人能够在无需特定于任务的训练的情况下完成新问题领域中的各种真实的机器人任务。SSR使用预训练的RL模型来创建表示演示和测试场景中的模型、任务和动作相关性的向量。然后，SSR通过我们的周期一致性距离（CCD）指标比较这些向量，以确定下一步要采取的行动。SSR完成的任务步骤比需要特定任务训练的基线少数学习方法多58%，试验次数多20%。与基线和从头开始训练RL模型相比，SSR还实现了计算效率的四个数量级改进和样本效率的20%到三个数量级改进。据我们所知，我们是第一个在真实的机器人上演示多步任务而无需特定任务训练的机器人，其中视觉输入和动作空间输出都是高维的。代码可在补充。
: Modern Reinforcement Learning (RL) algorithms are not sample-efficient to train on multi-step tasks in complex domains, impeding their wider deployment in the real world. We address this problem by leveraging the insight that RL models trained to complete one set of tasks can be re-purposed to complete related tasks when given just a handful of demonstrations. Based upon this insight, we propose See-SPOT-Run (SSR), a new computational approach to robot learning that enables a robot to complete a variety of real robot tasks in novel problem domains without task-specific training. SSR uses pretrained RL models to create vectors that represent model, task, and action relevance in demonstration and test scenes. SSR then compares these vectors via our Cycle Consistency Distance (CCD) metric to determine the next action to take. SSR completes 58% more task steps and 20% more trials than a baseline few-shot learning method that requires task-specific training. SSR also achieves a four order of magnitude improvement in compute efficiency and a 20% to three order of magnitude improvement in sample efficiency compared to the baseline and to training RL models from scratch. To our knowledge, we are the first to address multi-step tasks from demonstration on a real robot without task-specific training, where both the visual input and action space output are high dimensional. Code is available in the supplement.