Orchestrated Value Mapping for Reinforcement Learning
Orchestrated Value Mapping for Reinforcement Learning
复制标题
强化学习的精心编排的价值映射
DOI：
--
复制
发表时间：
2022
期刊：
影响因子：
--
通讯作者：
Arash Tavakoli
中科院分区：
文献类型：
--
作者：
Mehdi Fatemi;Arash Tavakoli
我们提出了一种通用的收敛类强化学习算法，该算法基于两个不同的原则：（1）使用广泛类别中的任意函数将值估计映射到不同的空间，以及（2）将奖励信号线性分解为多个通道。第一个原则可以将特定属性合并到可以增强学习的价值估计器中。另一方面，第二个原则允许将价值函数表示为多个效用函数的组合。这可以用于多种目的，例如处理高度变化的奖励尺度，结合有关奖励来源的先验知识和集成学习。结合这两个原则，通过在多个奖励通道上协调不同的映射函数，生成了实例化收敛算法的通用蓝图。该蓝图概括并包含了 Q-Learning、Log Q-Learning 和 Q-Decomposition 等算法。此外，我们对此类通用类的收敛证明放宽了其中一些算法中某些所需的假设。根据我们的理论，我们讨论了几种有趣的配置作为特殊情况。最后，为了说明我们的理论所开辟的设计空间的潜力，我们实例化了一个特定的算法并评估了它在 Atari 套件上的性能。
We present a general convergent class of reinforcement learning algorithms that is founded on two distinct principles: (1) mapping value estimates to a different space using arbitrary functions from a broad class, and (2) linearly decomposing the reward signal into multiple channels. The first principle enables incorporating specific properties into the value estimator that can enhance learning. The second principle, on the other hand, allows for the value function to be represented as a composition of multiple utility functions. This can be leveraged for various purposes, e.g. dealing with highly varying reward scales, incorporating a priori knowledge about the sources of reward, and ensemble learning. Combining the two principles yields a general blueprint for instantiating convergent algorithms by orchestrating diverse mapping functions over multiple reward channels. This blueprint generalizes and subsumes algorithms such as Q-Learning, Log Q-Learning, and Q-Decomposition. In addition, our convergence proof for this general class relaxes certain required assumptions in some of these algorithms. Based on our theory, we discuss several interesting configurations as special cases. Finally, to illustrate the potential of the design space that our theory opens up, we instantiate a particular algorithm and evaluate its performance on the Atari suite.
使用对数映射来降低强化学习中的折扣因子
DOI：
--
发表时间：
2019
期刊：
--
影响因子：
--
作者：
Van Seijen H
通讯作者：
Van Seijen H