Planning in Discrete and Continuous Markov Decision Processes by Probabilistic Programming
Planning in Discrete and Continuous Markov Decision Processes by Probabilistic Programming
复制标题
通过概率编程规划离散和连续马尔可夫决策过程
DOI：
--
复制
发表时间：
2015
期刊：
影响因子：
--
通讯作者：
L. D. Raedt
中科院分区：
文献类型：
--
作者：
D. Nitti;Vaishak Belle;L. D. Raedt
现实世界的规划问题经常涉及连续和离散状态变量和动作的混合，并且是在具有未知数量的对象的环境中制定的。近年来，概率规划已经成为一种自然的方法，可以用通用的推理方法来捕捉和刻画这种复杂的概率分布。众所周知，概率编程语言可以很容易地扩展为表示计划任务的马尔可夫决策过程(MDP)，但求解此类任务是具有挑战性的。在强化学习的相关工作的基础上，我们介绍了一种概念上简单但功能强大的MDP规划算法，该算法以概率规划的形式实现。该规划器在利用MDP模型知识的同时，通过重要性抽样来构造对最优策略的近似。实验结果表明，该方法在从严格离散到严格连续再到混合的领域都具有广泛的适用性，能够处理未知对象等复杂问题，并且具有较强的通用性。
Real-world planning problems frequently involve mixtures of continuous and discrete state variables and actions, and are formulated in environments with an unknown number of objects. In recent years, probabilistic programming has emerged as a natural approach to capture and characterize such complex probability distributions with general-purpose inference methods. While it is known that a probabilistic programming language can be easily extended to represent Markov Decision Processes (MDPs) for planning tasks, solving such tasks is challenging. Building on related efforts in reinforcement learning, we introduce a conceptually simple but powerful planning algorithm for MDPs realized as a probabilistic program. This planner constructs approximations to the optimal policy by importance sampling, while exploiting the knowledge of the MDP model. In our empirical evaluations, we show that this approach has wide applicability on domains ranging from strictly discrete to strictly continuous to hybrid ones, handles intricacies such as unknown objects, and is argued to be competitive given its generality.