Incremental State Aggregation for Value Function Estimation in Reinforcement Learning
Incremental State Aggregation for Value Function Estimation in Reinforcement Learning
复制标题
DOI：
10.1109/tsmcb.2011.2148710
复制
发表时间：
2011-10
期刊：
影响因子：
--
通讯作者：
Takeshi Mori;S. Ishii
中科院分区：
文献类型：
--
作者：
Takeshi Mori;S. Ishii
在强化学习中，大的状态和动作空间使得价值函数的估计不切实际，因此价值函数通常被表示为基函数的线性组合，其线性系数构成要估计的参数。然而，准备基函数需要一定的先验知识，并且通常是一项艰巨的任务。为了克服这个困难，Keller 最近提出了一种自适应基函数构造技术，但它需要过多的计算成本。我们提出了一种解决这一难题的有效方法，其中近似价值函数的问题被分解为许多子问题，每个子问题都可以用很小的计算成本来解决。计算机实验表明，我们的方法所需的CPU时间比现有方法小得多。
In reinforcement learning, large state and action spaces make the estimation of value functions impractical, so a value function is often represented as a linear combination of basis functions whose linear coefficients constitute parameters to be estimated. However, preparing basis functions requires a certain amount of prior knowledge and is, in general, a difficult task. To overcome this difficulty, an adaptive basis function construction technique has been proposed by Keller recently, but it requires excessive computational cost. We propose an efficient approach to this difficulty, in which the problem of approximating the value function is decomposed into a number of subproblems, each of which can be solved with small computational cost. Computer experiments show that the CPU time needed by our method is much smaller than that by the existing method.