Reinforcement Learning to Create Value and Policy Functions using Minimax Tree Search in Hex
Reinforcement Learning to Create Value and Policy Functions using Minimax Tree Search in Hex
复制标题
强化学习使用十六进制最小最大树搜索创建价值和策略函数
DOI：
10.1109/tg.2019.2893343
复制
发表时间：
2019
影响因子：
2.3
通讯作者：
Yamamoto Masahito
中科院分区：
文献类型：
--
作者：
Takada Kei;Iizuka Hiroyuki;Yamamoto Masahito
最近，已经提出使用学习算法来创建价值和策略函数，并且已经使用Go，Chess和Shogi证明了它们的有效性。在以前的研究中，策略函数被训练来预测蒙特卡洛树搜索输出的每个移动的搜索概率;因此，需要一些模拟来获得搜索概率。我们提出了一种基于自博弈的学习算法来创建值和策略函数，使得策略函数直接从博弈结果中训练而无需搜索概率。在这项研究中，我们使用十六进制，由Piet Hein开发的棋盘游戏，来评估所提出的方法。我们证明了所提出的学习算法在策略函数精度方面的有效性，并与所提出的计算机Hex算法DeepEZO和2017年世界冠军程序进行了比赛。比赛结果表明，DeepEZO优于所有程序。DeepEZO在13 × 13棋盘上，在相同的搜索条件下，对世界冠军程序MoHex2.0取得了79.3%的胜率。我们还表明，可以通过训练策略函数来创建高度准确的策略函数，以增加在失败者位置搜索的移动次数。
Recently, the use of reinforcement-learning algorithms has been proposed to create value and policy functions, and their effectiveness has been demonstrated using Go, Chess, and Shogi. In previous studies, the policy function was trained to predict the search probabilities of each move output by Monte Carlo tree search; thus, a number of simulations were required to obtain the search probabilities. We propose a reinforcement-learning algorithm with game of self-play to create value and policy functions such that the policy function is trained directly from the game results without the search probabilities. In this study, we use Hex, a board game developed by Piet Hein, to evaluate the proposed method. We demonstrate the effectiveness of the proposed learning algorithm in terms of the policy function accuracy, and play a tournament with the proposed computer Hex algorithm DeepEZO and 2017 world-champion programs. The tournament results demonstrate that DeepEZO outperforms all programs. DeepEZO achieved a winning percentage of 79.3% against the world-champion program MoHex2.0 under the same search conditions on 13 × 13 board. We also show that the highly accurate policy functions can be created by training the policy functions to increase the number of moves to be searched in the loser position.