How to train your robot with deep reinforcement learning: lessons we have learned
How to train your robot with deep reinforcement learning: lessons we have learned
复制标题
DOI：
10.1177/0278364920987859
复制
发表时间：
2021-04-01
影响因子：
9.2
通讯作者：
Levine, Sergey
中科院分区：
文献类型：
--
作者：
Ibarz, Julian;Tan, Jie;Levine, Sergey
深度强化学习（RL）已成为从低级传感器观察中自主获取复杂行为的一种有前途的方法。尽管深度强化学习的大部分研究都集中在视频游戏和模拟控制中的应用，这与现实环境中学习的限制无关，但深度强化学习也显示出使物理机器人能够在现实世界中学习复杂技能的前景。与此同时，现实世界的机器人技术为评估此类算法提供了一个有吸引力的领域，因为它直接与人类的学习方式相关：作为现实世界中的具体代理。学习在现实世界中感知和移动面临着许多挑战，其中一些挑战比其他挑战更容易解决，而其中一些挑战在仅关注模拟领域的强化学习研究中通常不会被考虑。在这篇评论文章中，我们介绍了一些涉及机器人深度强化学习的案例研究。在这些案例研究的基础上，我们讨论了深度强化学习中常见的挑战以及这些工作中如何解决这些挑战。我们还概述了其他突出的挑战，其中许多挑战是现实世界机器人环境所独有的，并且通常不是主流强化学习研究的焦点。我们的目标是为有兴趣推动深度强化学习在现实世界中取得进展的机器人专家和机器学习研究人员提供资源。
Deep reinforcement learning (RL) has emerged as a promising approach for autonomously acquiring complex behaviors from low-level sensor observations. Although a large portion of deep RL research has focused on applications in video games and simulated control, which does not connect with the constraints of learning in real environments, deep RL has also demonstrated promise in enabling physical robots to learn complex skills in the real world. At the same time, real-world robotics provides an appealing domain for evaluating such algorithms, as it connects directly to how humans learn: as an embodied agent in the real world. Learning to perceive and move in the real world presents numerous challenges, some of which are easier to address than others, and some of which are often not considered in RL research that focuses only on simulated domains. In this review article, we present a number of case studies involving robotic deep RL. Building off of these case studies, we discuss commonly perceived challenges in deep RL and how they have been addressed in these works. We also provide an overview of other outstanding challenges, many of which are unique to the real-world robotics setting and are not often the focus of mainstream RL research. Our goal is to provide a resource both for roboticists and machine learning researchers who are interested in furthering the progress of deep RL in the real world.