Planning in Stochastic Environments with a Learned Model
Planning in Stochastic Environments with a Learned Model
复制标题
使用学习模型在随机环境中进行规划
DOI：
--
复制
发表时间：
2022
期刊：
影响因子：
--
通讯作者：
David Silver
中科院分区：
文献类型：
--
作者：
Ioannis Antonoglou;Julian Schrittwieser;Sherjil Ozair;Thomas K. Hubert;David Silver
基于模型的强化学习已被证明非常成功。然而，在复杂环境中，脱离规划过程中的使用来学习模型是有问题的。到目前为止，最有效的技术是将价值等效模型学习与强大的树搜索方法相结合。MuZero就是这种方法的典范，它在从棋盘游戏到视觉丰富的环境，具有离散和连续动作空间，在线和离线设置等广泛领域都取得了最先进的性能。然而，这种方法之前的实例仅限于使用确定性模型。这限制了它们在本质上具有随机性、部分可观测，或者对有限智能体来说太大太复杂以至于看起来具有随机性的环境中的性能。在本文中，我们扩展了这种方法，以使用随机模型进行学习和规划。具体来说，我们引入了一种新算法——随机MuZero，它学习一个包含后状态的随机模型，并使用该模型进行随机树搜索。随机MuZero在一组标准的单智能体和多智能体环境中，包括2048和西洋双陆，达到或超过了现有技术水平，同时在围棋游戏中保持了标准MuZero的超人性能。
Model-based reinforcement learning has proven highly successful. However, learning a model in isolation from its use during planning is problematic in complex environments. To date, the most effective techniques have instead combined value-equivalent model learning with powerful tree-search methods. This approach is exemplified by MuZero , which has achieved state-of-the-art performance in a wide range of domains, from board games to visually rich environments, with discrete and continuous action spaces, in online and offline settings. However, previous instantiations of this approach were limited to the use of deterministic models. This limits their performance in environments that are inherently stochastic, partially observed, or so large and complex that they appear stochastic to a finite agent. In this paper we extend this approach to learn and plan with stochastic models. Specifically, we introduce a new algorithm, Stochastic MuZero , that learns a stochastic model incorporating afterstates, and uses this model to perform a stochastic tree search. Stochastic MuZero matched or exceeded the state of the art in a set of canonical single and multi-agent environments, including 2048 and backgammon, while maintaining the superhuman performance of standard MuZero in the game of Go.