Direct policy search as an alternative to POMDP for sequential decision problems in infrastructure planning
Direct policy search as an alternative to POMDP for sequential decision problems in infrastructure planning
复制标题
DOI：
10.22725/icasp13.212
复制
发表时间：
2019-05
期刊：
影响因子：
--
通讯作者：
E. Bismut;D. Štraub
中科院分区：
文献类型：
--
作者：
E. Bismut;D. Štraub
大多数基础设施规划挑战属于顺序决策问题，其特点是系统需求和性能的初始不确定性很大，以及在整个使用寿命期间收集信息和减少不确定性的可能性。本文考虑了一个通用的基础设施规划问题，并比较了两种主要的求解框架，部分可观测马尔可夫决策过程(POMDP)和直接策略搜索(DPS)以及启发式算法的选择。建立了一个案例研究，使得信念空间只用两个参数来描述，并且POMDP方法得到了精确的解。通过与POMDP解的比较，我们通过检验启发式的最优选择来研究直接策略搜索的性能。根据所考虑的系统和奖励函数的类型，定义试探法的参数可以是需求或系统可靠性的阈值，之后需要干预，或者是建议部件维修的临界损伤值。参数的选择直接影响所得解的优劣。我们为通用问题的启发式参数的最优选择确定关键因素，并提供对指导决策过程的系统的哪些特定功能的见解。
Most infrastructure planning challenges belong to the class of sequential decision problems, characterized by significant initial uncertainty on the demand and performance of the system, and the possibility to collect information and reduce the uncertainty throughout the service life. In this paper, we consider a generic infrastructure planning problem and compare two main solution frameworks, partially observable Markov decision processes (POMDPs) and direct policy search (DPS) with a choice of heuristics. A case study is set up so that the the belief space is described by only two parameters and the POMDP approach yields an exact solution. We investigate the performance of direct policy search by examining the optimal choice of the heuristics through a comparison with the POMDP solution. Depending on the type of system and reward function considered, parameters defining the heuristics can be thresholds on the demand or the system reliability, after which intervention is required, or critical damage values that suggest a component repair. The choice of the parameters directly influences the goodness of the solution found. We identify key factors for the optimal selection of the heuristic parameters for the generic problem, and provide insights into which specific features of the system that guide the decision process.