Monte Carlo tree search with optimal computing budget allocation
Monte Carlo tree search with optimal computing budget allocation
复制标题
具有最佳计算预算分配的蒙特卡洛树搜索
DOI：
10.1109/cdc40024.2019.9030099
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
Jie Xu
中科院分区：
文献类型：
--
作者：
Yunchuan Li;M. Fu;Jie Xu
我们分析了一个基于马尔可夫决策过程的树搜索问题，在该问题中，目标是确定在根部获得最高累积回报的最佳动作。我们提出了一种新的树策略，它最优地分配有限的计算预算，以最大化在每个节点正确选择最佳动作的概率的下界。与广泛使用的上置信度(UCB)型树策略相比，新的树策略在采样预算有限的情况下提供了一种更平衡的方法来管理勘探和开采之间的权衡。此外，UCB假设报酬分布的支持度是已知的，而我们的算法放宽了这一假设，并可以稍作修改就可以应用于博弈树。通过数值实验验证了该算法在选择最优根部动作方面的有效性。
We analyze a tree search problem with an underlying Markov decision process, in which the goal is to identify the best action at the root that achieves the highest cumulative reward. We present a new tree policy that optimally allocates a limited computing budget to maximize a lower bound on the probability of correctly selecting the best action at each node. Compared to the widely used Upper Confidence Bound (UCB) type of tree policies, the new tree policy presents a more balanced approach to manage the exploration and exploitation trade-off when the sampling budget is limited. Furthermore, UCB assumes that the support of reward distribution is known, whereas our algorithm relaxes this assumption, and can be applied to game trees with mild modifications. A numerical experiment is conducted to demonstrate the efficiency of our algorithm in selecting the best action at the root.