Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learning
Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learning
复制标题
DOI：
--
复制
发表时间：
2021-11
期刊：
影响因子：
--
通讯作者：
Yingjie Fei;Zhuoran Yang;Yudong Chen;Zhaoran Wang
中科院分区：
文献类型：
--
作者：
Yingjie Fei;Zhuoran Yang;Yudong Chen;Zhaoran Wang
研究了基于熵风险测度的风险敏感强化学习。虽然现有的作品已经建立了非渐近后悔保证这个问题，他们留下了开放的上界和下界之间的指数差距。我们确定在现有的算法和他们的分析，导致这样的差距的不足之处。为了弥补这些不足，我们研究了一个简单的转换的风险敏感的贝尔曼方程，我们称之为指数贝尔曼方程。指数Bellman方程启发了我们对风险敏感RL算法中的Bellman备份过程进行新的分析，并进一步激发了新探索机制的设计。我们表明，这些分析和算法的创新共同导致改进的遗憾上限比现有的。
We study risk-sensitive reinforcement learning (RL) based on the entropic risk measure. Although existing works have established non-asymptotic regret guarantees for this problem, they leave open an exponential gap between the upper and lower bounds. We identify the deficiencies in existing algorithms and their analysis that result in such a gap. To remedy these deficiencies, we investigate a simple transformation of the risk-sensitive Bellman equations, which we call the exponential Bellman equation. The exponential Bellman equation inspires us to develop a novel analysis of Bellman backup procedures in risk-sensitive RL algorithms, and further motivates the design of a novel exploration mechanism. We show that these analytic and algorithmic innovations together lead to improved regret upper bounds over existing ones.