A theoretical framework for probabilistic reinforcement learning in the basal ganglia

基底神经节概率强化学习的理论框架

• 批准号: 10460155
• 负责人: Samuel J Gershman
• 金额: $ 53.39万
• 依托单位: HARVARD MEDICAL SCHOOL
• 依托单位国家: 美国
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-08-15 至 2024-07-31
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/10460155
• 关键词: Adaptive Behaviors; Address; Animals; Architecture; Basal Ganglia; Behavior; Belief; Cells; Corpus striatum structure; Data; Data Analyses; Dopamine; Dorsal; Experimental Designs; Future; Goals; Learning; Link; Location; Modeling; Neurons; Output; Pathway interactions; Pattern; Play; Policies; Probability; Psychological reinforcement; Ramp; Rattus; Rewards; Rodent; Role; Signal Transduction; Specific qualifier value; Synapses; Testing; Time; Uncertainty; Update; Weight; Width; Work; base; design; flexibility; innovation; insight; mathematical model; motor behavior; neurobiological mechanism; predictive modeling; success; theories

Project abstract According to the standard reinforcement learning framework, the basal ganglia implements estimation of long- term future reward and the control of actions to maximize future reward. Dopamine (DA) plays a central role by providing the learning signal (reward prediction error, or RPE) that guides updating of reward predictions and the action policy. Despite its success, the reinforcement learning framework has been challenged from a number of directions. Some studies have suggested that DA encodes reward predictions themselves, rather than reward prediction errors, and other studies have suggested that DA may play a role in invigorating action selection independently from its contribution to learning. A major goal of this project is to develop a reinforcement learning theory of basal ganglia function that addresses these challenges, and more broadly presents a unifying view of how learning, probabilistic inference, and action selection work together to produce adaptive behavior. Our theoretical innovation can be divided into three components. First, we argue that cortical inputs to the striatum encode a probability distribution over hidden states, known as the belief state. Second, we argue that striatal projection neurons transform this input through a set of basis functions, whose purpose is to facilitate reward prediction. The synaptic weights that parametrize these predictions are updated based on the DA RPE signal. Third, we argue that action selection circuits in the dorsal striatum use probabilistic information about rewards to implement uncertainty-guided exploration.

项目摘要 根据标准增强学习框架，基底神经节实施了长期的估计 术语未来的奖励和对最大化未来奖励的行动的控制。多巴胺（DA）扮演着核心角色 提供指导更新奖励预测和的学习信号（奖励预测错误或RPE） 行动政策。尽管取得了成功，但增强学习框架已从 方向数。一些研究表明，DA本身编码奖励预测，而是 而不是奖励预测错误，其他研究表明，DA可能在振兴行动中发挥作用 选择独立于其对学习的贡献。该项目的主要目标是开发 基础神经节功能的强化学习理论解决这些挑战，更广泛地解决这些挑战 对学习，概率推理和行动选择如何共同生产的统一观点 自适应行为。我们的理论创新可以分为三个组成部分。首先，我们认为 纹状体的皮质输入对隐藏状态（称为信仰状态）的概率分布编码。 其次，我们认为纹状体投影神经元通过一组基础函数转化了此输入， 目的是促进奖励预测。参数化这些预测的突触权重更新了 基于DA RPE信号。第三，我们认为背纹状体中的动作选择电路 有关奖励的概率信息，以实施不确定性引导的探索。