Optimistic Exploration even with a Pessimistic Initialisation
Optimistic Exploration even with a Pessimistic Initialisation
复制标题
DOI：
--
复制
发表时间：
2020-02
期刊：
影响因子：
--
通讯作者：
Tabish Rashid;Bei Peng;Wendelin Böhmer;Shimon Whiteson
中科院分区：
文献类型：
--
作者：
Tabish Rashid;Bei Peng;Wendelin Böhmer;Shimon Whiteson
乐观初始化是强化学习（RL）中有效探索的一种有效策略。在表格的情况下，所有可证明有效的无模型算法都依赖于它。然而，无模型深度强化学习算法并不使用乐观初始化，尽管它们从这些可证明有效的表格算法中获得了灵感。特别是，在只有正奖励的情况下，由于常用的网络初始化方案，Q值被初始化为它们的最低可能值，这是一种悲观的初始化。仅仅初始化网络以输出乐观的Q值是不够的，因为我们不能确保它们对新的状态-动作对保持乐观，这对探索至关重要。我们提出了一个简单的基于计数的增强悲观初始化的Q值，从神经网络中分离乐观的来源。我们证明了该方案在表格设置中是可证明有效的，并将其扩展到深度RL设置。我们的算法，乐观悲观初始化Q学习（OPIQ），增加了Q值估计的DQN为基础的代理与计数派生奖金，以确保在行动选择和引导过程中的乐观。我们表明，OPIQ优于非乐观的DQN变种，利用伪计数为基础的内在动机在硬探索任务，它预测乐观的估计新的状态-动作对。
Optimistic initialisation is an effective strategy for efficient exploration in reinforcement learning (RL). In the tabular case, all provably efficient model-free algorithms rely on it. However, model-free deep RL algorithms do not use optimistic initialisation despite taking inspiration from these provably efficient tabular algorithms. In particular, in scenarios with only positive rewards, Q-values are initialised at their lowest possible values due to commonly used network initialisation schemes, a pessimistic initialisation. Merely initialising the network to output optimistic Q-values is not enough, since we cannot ensure that they remain optimistic for novel state-action pairs, which is crucial for exploration. We propose a simple count-based augmentation to pessimistically initialised Q-values that separates the source of optimism from the neural network. We show that this scheme is provably efficient in the tabular setting and extend it to the deep RL setting. Our algorithm, Optimistic Pessimistically Initialised Q-Learning (OPIQ), augments the Q-value estimates of a DQN-based agent with count-derived bonuses to ensure optimism during both action selection and bootstrapping. We show that OPIQ outperforms non-optimistic DQN variants that utilise a pseudocount-based intrinsic motivation in hard exploration tasks, and that it predicts optimistic estimates for novel state-action pairs.