Reinforcement learning with incremental skill models: Extension to tool use as skills
Reinforcement learning with incremental skill models: Extension to tool use as skills
复制标题
DOI：
10.1109/sii55687.2023.10039431
复制
发表时间：
2023-01
期刊：
影响因子：
--
通讯作者：
Ryota Yashima;Akihiko Yamaguchi;Koichi Hashimoto
中科院分区：
文献类型：
--
作者：
Ryota Yashima;Akihiko Yamaguchi;Koichi Hashimoto
考虑到对各种工具和对象的操作，例如机器人的烹饪任务，人们提出了各种获取行为的方法，但机器人获取的行为知识的重用和获取通用知识仍然是一个难题。在我们使用技能库的基于模型的强化学习中，机器人的行为被表示为技能图，并且每种技能的动态建模用于运动规划。该方法的优点是学习的动力学模型可以在其他任务和情况下重用，我们相信这将是通用机器人运动规划方法的基础。在使用技能库的基于模型的强化学习中，没有考虑在现有技能库中引入工具。将工具使用技能添加到技能库中，我们认为可解决的情况将比以往任何时候都要多。例如，如果我们在液体浇注任务中引入漏斗，我们可以改变液体流动的动力学，使其减少溢出。另一方面，从动力学模型设计的角度来看，以一种不受漏斗使用影响的方式建模流动力学在可重用性方面是可取的。在本研究中，我们将挤压技能（用软容器挤出）和安装漏斗技能（用漏斗扩大接收口）引入液体浇注任务，作为使用基于模型的强化学习的运动规划案例研究，使用技能库引入工具技能。我们设计了不依赖于工具添加的动力学模型，并表明机器人根据情况获得适当使用工具的运动。我们还分析了学习工具使用技能动态模型与其他训练过的动态模型相结合的效果。
Considering manipulation of various tools and objects, such as the cooking task of robots, various methods of acquiring behaviors have been proposed, but reusing the behavioral knowledge acquired by robots and acquiring generic knowledge remain difficult issues. In our model-based reinforcement learning using the skill library, the behavior of the robot is represented as a graph of skills, and the dynamics of each skill are modeled for motion planning. This method has the advantage that the learned dynamics model can be reused in other tasks and situations, and we believe that this will be the basis for a general-purpose robot motion planning method. In model-based reinforcement learning using the skill library, the introduction of tools into the existing skill library has not been considered. Adding tool-use skills into the skill library, we think resolvable situations will expand more than ever. For example, if we introduce a funnel into the liquid pouring task, we can change the dynamics of the liquid flow to make it less spillage. On the other hand, from the viewpoint of dynamics model design, modeling the flow dynamics in a way that is not affected by the use of funnels is preferable in terms of reusability. In this study, we introduce Squeeze skill (squeeze out with a soft container) and Mount Funnel skill (enlarge the receiving mouth with a funnel) into the liquid pouring task as a case study of motion planning using model-based reinforcement learning using the skill library introduced tool skills. We design dynamics models that do not depend on the addition of tools and show that the robot acquires motions that appropriately use tools according to the situation. We also analyze the effect of learning a tool-use skill dynamics model in combination with other trained dynamics models.