Planning with Abstract Markov Decision Processes
Planning with Abstract Markov Decision Processes
复制标题
DOI：
10.1609/icaps.v27i1.13867
复制
发表时间：
2017-06
期刊：
影响因子：
--
通讯作者：
N. Gopalan;Marie desJardins;M. Littman;J. MacGlashan;S. Squire;Stefanie Tellex;J. Winder;Lawson L. S. Wong
中科院分区：
文献类型：
--
作者：
N. Gopalan;Marie desJardins;M. Littman;J. MacGlashan;S. Squire;Stefanie Tellex;J. Winder;Lawson L. S. Wong
在人类规模的环境中行动的机器人必须在大的状态-动作空间中的不确定性下进行规划，并且随着需求和目标的变化而面临不断变化的奖励函数。在大的状态-动作空间中的不确定性下的规划需要分层抽象以实现有效的计算。我们引入了一个新的层次规划框架，称为抽象马尔可夫决策过程（AMDPs），可以计划在一小部分的时间需要复杂的决策，在普通的MDP。AMDPs提供抽象的状态，动作，和过渡动态在多个层次以上的基本级别的“平面”MDP。AMDPs将问题分解为一系列子任务，并使用局部奖励和局部转换函数为子任务创建策略。由此产生的分层规划方法是独立的最佳在每个抽象层次，是递归最佳的地方奖励和过渡函数是正确的。我们目前的实证结果显示显着提高规划速度，同时保持解决方案的质量，在出租车领域和移动操作机器人问题。此外，我们的方法允许规范的决策模型的移动操作问题的海龟机器人，跨越从低层次的控制操作连续变量的所有方式通过高层次的对象操作任务。
Robots acting in human-scale environments must plan under uncertainty in large state–action spaces and face constantly changing reward functions as requirements and goals change. Planning under uncertainty in large state–action spaces requires hierarchical abstraction for efficient computation. We introduce a new hierarchical planning framework called Abstract Markov Decision Processes (AMDPs) that can plan in a fraction of the time needed for complex decision making in ordinary MDPs. AMDPs provide abstract states, actions, and transition dynamics in multiple layers above a base-level “flat” MDP. AMDPs decompose problems into a series of subtasks with both local reward and local transition functions used to create policies for subtasks. The resulting hierarchical planning method is independently optimal at each level of abstraction, and is recursively optimal when the local reward and transition functions are correct. We present empirical results showing significantly improved planning speed, while maintaining solution quality, in the Taxi domain and in a mobile-manipulation robotics problem. Furthermore, our approach allows specification of a decision-making model for a mobile-manipulation problem on a Turtlebot, spanning from low-level control actions operating on continuous variables all the way up through high-level object manipulation tasks.