Reinforcement Learning for Constrained Markov Decision Processes
Reinforcement Learning for Constrained Markov Decision Processes
复制标题
约束马尔可夫决策过程的强化学习
DOI：
--
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
V. Aggarwal
中科院分区：
文献类型：
--
作者：
Ather Gattami;Qinbo Bai;V. Aggarwal
在本文中，我们考虑的问题的优化和学习的约束和多目标马尔可夫决策过程，折扣奖励和期望平均奖励。我们制定的零和游戏的问题，其中一个球员（代理）解决了马尔可夫决策问题，其对手解决了强盗优化问题，我们在这里称之为马尔可夫-强盗游戏。我们将Q -学习扩展到解决Markov-Bandit博弈，并证明了我们的新Q -学习算法收敛到零和Markov-Bandit博弈的最优解，从而收敛到约束和多目标马尔可夫决策问题的最优解。我们提供了数值例子，我们计算出的最优策略，并通过模拟表明，该算法收敛到计算出的最优策略。据我们所知，这是Q学习算法第一次保证收敛到分别具有折扣和预期平均奖励的多目标强化学习问题的最佳静态策略。
In this paper, we consider the problem of optimization and learning for constrained and multi-objective Markov decision processes, for both discounted rewards and expected average rewards. We formulate the problems as zero-sum games where one player (the agent) solves a Markov decision problem and its opponent solves a bandit optimization problem, which we here call Markov-Bandit games. We extend Q -learning to solve Markov-Bandit games and show that our new Q -learning algorithms converge to the optimal solutions of the zero-sum Markov-Bandit games, and hence converge to the optimal solutions of the constrained and multi-objective Markov decision problems. We provide numerical examples where we calculate the optimal policies and show by simulations that the algorithm converges to the calculated optimal policies. To the best of our knowledge, this is the first time Q-learning algorithms guarantee convergence to optimal stationary policies for the multi-objective Reinforcement Learning problem with discounted and expected average rewards, respectively.
DOI：
--
发表时间：
2020-03
期刊：
--
影响因子：
--
作者：
Dongsheng Ding;Xiaohan Wei;Zhuoran Yang;Zhaoran Wang;M. Jovanovi'c
通讯作者：
Dongsheng Ding;Xiaohan Wei;Zhuoran Yang;Zhaoran Wang;M. Jovanovi'c