Scaling Active Inference
Scaling Active Inference
复制标题
DOI：
10.1109/ijcnn48605.2020.9207382
复制
发表时间：
2019-11
期刊：
影响因子：
--
通讯作者：
Alexander Tschantz;Manuel Baltieri;A. Seth;C. Buckley
中科院分区：
文献类型：
--
作者：
Alexander Tschantz;Manuel Baltieri;A. Seth;C. Buckley
在强化学习（RL）中，智能体通常在部分观察和不确定的环境中运行。基于模型的强化学习表明，这是通过学习和利用世界的概率模型来实现的。“主动推理”是认知和计算神经科学中一个新兴的规范框架，它提供了生物因子如何实现这一目标的统一解释。在这个框架下，推理、学习和行动都来自于一个单一的命令，即最大化贝叶斯证据，为一个小众的世界模型提供证据。然而，到目前为止，该过程的实现仅限于低维和理想化的情况。在这里，我们提出了一个适用于高维任务的主动推理的工作实现，其原理证明结果显示了有效的探索，并且在强无模型基线上的样本效率提高了一个数量级。我们的研究结果证明了大规模应用主动推理的可行性，并强调了主动推理与当前基于模型的强化学习方法之间的操作同源性。
In reinforcement learning (RL), agents often operate in partially observed and uncertain environments. Model-based RL suggests that this is best achieved by learning and exploiting a probabilistic model of the world. ‘Active inference’ is an emerging normative framework in cognitive and computational neuroscience that offers a unifying account of how biological agents achieve this. On this framework, inference, learning and action emerge from a single imperative to maximize the Bayesian evidence for a niched model of the world. However, implementations of this process have thus far been restricted to low-dimensional and idealized situations. Here, we present a working implementation of active inference that applies to high-dimensional tasks, with proof-of-principle results demonstrating efficient exploration and an order of magnitude increase in sample efficiency over strong model-free baselines. Our results demonstrate the feasibility of applying active inference at scale and highlight the operational homologies between active inference and current model-based approaches to RL.