A theoretical framework for probabilistic reinforcement learning in the basal ganglia

项目摘要 根据标准的强化学习框架，基底神经节实现长- 长期未来奖励和控制行动以最大化未来奖励。多巴胺（DA）通过以下途径发挥核心作用： - 提供引导奖励预测的更新的学习信号（奖励预测误差，或RPE），以及 行动政策。尽管取得了成功，但强化学习框架受到了来自 方向的数量。一些研究表明，DA编码奖励预测本身，而不是 而不是奖励预测错误，其他研究表明，DA可能在激励行动中发挥作用， 选择独立于其对学习的贡献。该项目的一个主要目标是开发一个 基底神经节功能的强化学习理论解决了这些挑战， 提出了一个统一的观点，学习，概率推理和行动选择如何协同工作，以产生 适应行为我们的理论创新可以分为三个组成部分。首先，我们认为， 对纹状体的皮层输入编码了被称为信念状态的隐藏状态的概率分布。 其次，我们认为纹状体投射神经元通过一组基函数来转换这种输入， 目的是为了便于奖励预测。更新参数化这些预测的突触权重 基于DA RPE信号。第三，我们认为，动作选择回路在背侧纹状体使用， 关于奖励的概率信息，以实现不确定性引导的探索。

Project abstract According to the standard reinforcement learning framework, the basal ganglia implements estimation of long- term future reward and the control of actions to maximize future reward. Dopamine (DA) plays a central role by providing the learning signal (reward prediction error, or RPE) that guides updating of reward predictions and the action policy. Despite its success, the reinforcement learning framework has been challenged from a number of directions. Some studies have suggested that DA encodes reward predictions themselves, rather than reward prediction errors, and other studies have suggested that DA may play a role in invigorating action selection independently from its contribution to learning. A major goal of this project is to develop a reinforcement learning theory of basal ganglia function that addresses these challenges, and more broadly presents a unifying view of how learning, probabilistic inference, and action selection work together to produce adaptive behavior. Our theoretical innovation can be divided into three components. First, we argue that cortical inputs to the striatum encode a probability distribution over hidden states, known as the belief state. Second, we argue that striatal projection neurons transform this input through a set of basis functions, whose purpose is to facilitate reward prediction. The synaptic weights that parametrize these predictions are updated based on the DA RPE signal. Third, we argue that action selection circuits in the dorsal striatum use probabilistic information about rewards to implement uncertainty-guided exploration.