Quantile QT-Opt for Risk-Aware Vision-Based Robotic Grasping
Quantile QT-Opt for Risk-Aware Vision-Based Robotic Grasping
复制标题
Quantile QT-Opt 用于基于风险意识的视觉机器人抓取
DOI：
--
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
Mrinal Kalakrishnan
中科院分区：
文献类型：
--
作者：
Cristian Bodnar;A. Li;Karol Hausman;P. Pastor;Mrinal Kalakrishnan
强化学习（RL）的分布式视角已经产生了一系列成功的Q学习算法，从而在街机游戏环境中实现了最先进的性能。然而，它还没有被分析如何将这些发现从一个离散的设置转化为复杂的实际应用，其特点是嘈杂的，高维的和连续的状态动作空间。在这项工作中，我们提出了分位数QT-选择（Q2-Opt），最近推出的分布式Q-学习算法的连续域的分布变体，并检查其行为在一系列模拟和真实的基于视觉的机器人抓取任务。在Q2-Opt中没有演员，这使得我们可以直接与文献中以前的离散实验进行比较，而无需演员-评论家架构引起的额外复杂性。我们证明了Q2-Opt实现了上级基于视觉的物体抓取成功率，同时也具有更高的采样效率。分布式公式还允许我们实验各种风险失真指标，这些指标为我们提供了机器人如何使用深度RL控制策略在实践中具体管理风险的指示。作为额外的贡献，我们在虚拟环境中进行批量RL实验，并将其与离散设置的最新发现进行比较。令人惊讶的是，我们发现，从街机游戏环境中获得的文献中先前的批次RL结果并不适用于我们的设置。
The distributional perspective on reinforcement learning (RL) has given rise to a series of successful Q-learning algorithms, resulting in state-of-the-art performance in arcade game environments. However, it has not yet been analyzed how these findings from a discrete setting translate to complex practical applications characterized by noisy, high dimensional and continuous state-action spaces. In this work, we propose Quantile QT-Opt (Q2-Opt), a distributional variant of the recently introduced distributed Q-learning algorithm for continuous domains, and examine its behaviour in a series of simulated and real vision-based robotic grasping tasks. The absence of an actor in Q2-Opt allows us to directly draw a parallel to the previous discrete experiments in the literature without the additional complexities induced by an actor-critic architecture. We demonstrate that Q2-Opt achieves a superior vision-based object grasping success rate, while also being more sample efficient. The distributional formulation also allows us to experiment with various risk distortion metrics that give us an indication of how robots can concretely manage risk in practice using a Deep RL control policy. As an additional contribution, we perform batch RL experiments in our virtual environment and compare them with the latest findings from discrete settings. Surprisingly, we find that the previous batch RL findings from the literature obtained on arcade game environments do not generalise to our setup.