Approximate Policy Iteration with a Policy Language Bias: Solving Relational Markov Decision Processes
Approximate Policy Iteration with a Policy Language Bias: Solving Relational Markov Decision Processes
复制标题
具有策略语言偏差的近似策略迭代：解决关系马尔可夫决策过程
DOI：
--
复制
发表时间：
2011
影响因子：
5
通讯作者：
S. Yoon
中科院分区：
文献类型：
--
作者：
Alan Fern;R. Givan;S. Yoon
我们研究了大型关系马尔可夫决策过程（MDP）的策略选择方法。我们考虑近似策略迭代（API）的一种变体，它用策略空间中的学习步骤代替通常的价值函数学习步骤。这在好的策略比相应的价值函数更容易表示和学习的领域是有利的，这对于我们感兴趣的关系 MDP 来说通常是这种情况。为了将 API 应用于此类问题，我们引入了关系策略语言和相应的学习器。此外，我们还为基于目标的规划领域引入了一种新的基于随机游走的引导例程。这种引导对于许多大型关系型 MDP 来说是必要的，因为在这些领域中，奖励极其稀疏，因为当使用不知情的策略进行初始化时，API 在此类领域中是无效的。我们的实验表明，最终的系统能够通过将许多经典规划领域及其随机变体解决为极大的关系 MDP 来找到良好的策略。实验还指出了我们方法的一些局限性，并提出了未来的工作建议。
We study an approach to policy selection for large relational Markov Decision Processes (MDPs). We consider a variant of approximate policy iteration (API) that replaces the usual value-function learning step with a learning step in policy space. This is advantageous in domains where good policies are easier to represent and learn than the corresponding value functions, which is often the case for the relational MDPs we are interested in. In order to apply API to such problems, we introduce a relational policy language and corresponding learner. In addition, we introduce a new bootstrapping routine for goal-based planning domains, based on random walks. Such bootstrapping is necessary for many large relational MDPs, where reward is extremely sparse, as API is ineffective in such domains when initialized with an uninformed policy. Our experiments show that the resulting system is able to find good policies for a number of classical planning domains and their stochastic variants by solving them as extremely large relational MDPs. The experiments also point to some limitations of our approach, suggesting future work.