Opponent Actor Learning (OpAL): Modeling Interactive Effects of Striatal Dopamine on Reinforcement Learning and Choice Incentive
Opponent Actor Learning (OpAL): Modeling Interactive Effects of Striatal Dopamine on Reinforcement Learning and Choice Incentive
复制标题
DOI：
10.1037/a0037015
复制
发表时间：
2014-07-01
影响因子：
5.4
通讯作者：
Frank, Michael J.
中科院分区：
文献类型：
--
作者：
Collins, Anne G. E.;Frank, Michael J.
纹状体多巴胺能系统与强化学习（RL），运动表现和激励动机有关。已经提出了各种计算模型来分别考虑这些影响，但缺乏对它们之间相互作用的正式分析。在这里，我们提出了一种新的算法模型，扩展了经典的演员-评论家架构，包括神经电路模型的基本交互特性，将激励和学习效果纳入一个单一的理论框架。标准的演员被取代的双对手演员系统，代表不同的纹状体人口，这来区别专门区分积极和消极的行动价值观。多巴胺调节每个参与者成分对学习和选择辨别的贡献程度。与标准框架相比，该模型同时捕获了多巴胺对学习和选择动机及其相互作用的影响，这些影响跨越了各种研究，包括概率RL，基于努力的选择和运动技能学习。
The striatal dopaminergic system has been implicated in reinforcement learning (RL), motor performance, and incentive motivation. Various computational models have been proposed to account for each of these effects individually, but a formal analysis of their interactions is lacking. Here we present a novel algorithmic model expanding the classical actor-critic architecture to include fundamental interactive properties of neural circuit models, incorporating both incentive and learning effects into a single theoretical framework. The standard actor is replaced by a dual opponent actor system representing distinct striatal populations, which come to differentially specialize in discriminating positive and negative action values. Dopamine modulates the degree to which each actor component contributes to both learning and choice discriminations. In contrast to standard frameworks, this model simultaneously captures documented effects of dopamine on both learning and choice incentive-and their interactions-across a variety of studies, including probabilistic RL, effort-based choice, and motor skill learning.