Munchausen Reinforcement Learning
Munchausen Reinforcement Learning
复制标题
DOI：
--
复制
发表时间：
2020-07
期刊：
影响因子：
--
通讯作者：
Nino Vieillard;O. Pietquin;M. Geist
中科院分区：
文献类型：
--
作者：
Nino Vieillard;O. Pietquin;M. Geist
自举是强化学习的核心机制。大多数算法基于时间差异，将过渡状态的真实值替换为当前对该值的估计。然而，另一个估计可以被用来引导RL：当前的政策。我们的核心贡献在于一个非常简单的想法：将按比例调整的原木政策添加到直接奖励中。我们证明了以这种方式略微修改Deep Q-Network(DQN)可以提供一个与Atari游戏上的分发方法竞争的代理，而不使用分发RL、n步返回或优先重放。为了展示这个想法的多功能性，我们还将其与隐式分位数网络(IQN)一起使用。生成的代理在Atari上的性能优于彩虹，安装了一个新的技术状态，对原始算法进行了很小的修改。为了补充这一实证研究，我们提供了关于引擎盖下发生的事情的强有力的理论见解--隐式Kullback-Leibler正则化和行动差距的增加。
Bootstrapping is a core mechanism in Reinforcement Learning (RL). Most algorithms, based on temporal differences, replace the true value of a transiting state by their current estimate of this value. Yet, another estimate could be leveraged to bootstrap RL: the current policy. Our core contribution stands in a very simple idea: adding the scaled log-policy to the immediate reward. We show that slightly modifying Deep Q-Network (DQN) in that way provides an agent that is competitive with distributional methods on Atari games, without making use of distributional RL, n-step returns or prioritized replay. To demonstrate the versatility of this idea, we also use it together with an Implicit Quantile Network (IQN). The resulting agent outperforms Rainbow on Atari, installing a new State of the Art with very little modifications to the original algorithm. To add to this empirical study, we provide strong theoretical insights on what happens under the hood -- implicit Kullback-Leibler regularization and increase of the action-gap.