Sigmoid-weighted linear units for neural network function approximation in reinforcement learning
Sigmoid-weighted linear units for neural network function approximation in reinforcement learning
复制标题
DOI：
10.1016/j.neunet.2017.12.012
复制
发表时间：
2018-11-01
期刊：
影响因子：
7.8
通讯作者：
Doya, Kenji
中科院分区：
文献类型：
--
作者：
Elfwing, Stefan;Uchibe, Eiji;Doya, Kenji
近年来，神经网络作为函数逼近器在强化学习中得到了复兴。在Tesauro的TD-Gammon在西洋双陆棋中取得接近人类顶级水平的成绩20年后，深度强化学习算法DQN在许多Atari 2600游戏中取得了人类水平的成绩。这项研究的目的是双重的。首先，我们提出了用于强化学习中神经网络函数逼近的两个激活函数：s型加权线性单元（SiLU）及其导数函数（dSiLU）。SiLU的激活是通过sigmoid函数乘以它的输入来计算的。其次，我们建议使用带有资格跟踪的策略学习（而不是经验回放）和softmax动作选择的更传统的方法可以与DQN竞争，而不需要单独的目标网络。我们验证了我们提出的方法，首先，使用TD（lambda）学习和浅层dSiLU网络代理，在随机SZ-Tetris和带有小10 x 10棋盘的俄罗斯方块中获得了新的最先进的结果，然后，通过使用带有SiLU和dSiLU隐藏单元的深度Sarsa（lambda）代理，在Atari 2600域中优于DQN。(C) 2017作者。Elsevier Ltd.出版。
In recent years, neural networks have enjoyed a renaissance as function approximators in reinforcement learning. Two decades after Tesauro's TD-Gammon achieved near top-level human performance in backgammon, the deep reinforcement learning algorithm DQN achieved human-level performance in many Atari 2600 games. The purpose of this study is twofold. First, we propose two activation functions for neural network function approximation in reinforcement learning: the sigmoid-weighted linear unit (SiLU) and its derivative function (dSiLU). The activation of the SiLU is computed by the sigmoid function multiplied by its input. Second, we suggest that the more traditional approach of using on-policy learning with eligibility traces, instead of experience replay, and softmax action selection can be competitive with DQN, without the need for a separate target network. We validate our proposed approach by, first, achieving new state-of-the-art results in both stochastic SZ-Tetris and Tetris with a small 10 x 10 board, using TD(lambda) learning and shallow dSiLU network agents, and, then, by outperforming DQN in the Atari 2600 domain by using a deep Sarsa(lambda) agent with SiLU and dSiLU hidden units. (C) 2017 The Author(s). Published by Elsevier Ltd.