Multiple memory systems as substrates for multiple decision systems.
Multiple memory systems as substrates for multiple decision systems.
复制标题
DOI：
10.1016/j.nlm.2014.04.014
复制
发表时间：
2015-01
影响因子：
2.7
通讯作者：
Daw, Nathaniel D.
中科院分区：
文献类型：
--
作者：
Doll, Bradley B.;Shohamy, Daphna;Daw, Nathaniel D.
最近，人们广泛认识到，基于价值的决策是由多种计算策略支持的。特别是，动物和人类在学习任务中的行为似乎包括由著名的无模型强化学习（RL）理论描述的习惯性反应，但也有更多的审议或目标导向的行动，可以通过另一类理论，基于模型的强化学习来表征。后一种理论通过使用任务的偶然性表示（如空间迷宫的学习地图）来评估行为，称为“内部模型”。鉴于这些方法之间存在行为和神经分离的证据，它们通常被描述为可分离的学习系统，尽管它们可能相互作用并共享共同的机制。在许多方面，这种划分类似于认知神经科学中多个记忆系统之间长期存在的分离，在最广泛的层面上描述了陈述性和程序性学习的独立系统。程序学习与无模型强化学习有显著的相似之处：两者都涉及习惯的学习，并且都依赖于纹状体的部分。相比之下，陈述性记忆支持对单个事件或情节的记忆，并依赖于海马体。海马体被认为通过编码刺激之间的时间和空间关系来支持陈述性记忆，因此通常被称为关系记忆系统。这种关系编码可能在学习内部模型中发挥重要作用，而内部模型是基于模型的强化学习的核心。因此，就记忆系统所代表的更通用的认知机制而言，这些认知机制可能会支持包括决策在内的许多任务的表现，这些相似之处提出了一个问题，即多个记忆系统是否服务于多个决策系统，这样一种分离是建立在另一种分离的基础上的。在这里，我们通过比较不同行为任务的个体差异来研究基于模型的强化学习和关系记忆之间的关系。人类受试者执行两项任务，一项是学习和泛化任务（习得对等），该任务涉及关系编码并依赖于海马体；以及可以通过基于模型或无模型策略解决的顺序RL任务。我们评估了被试在习得性对等任务中使用灵活、关系记忆（通过泛化来衡量）与他们在决策任务中对两种强化学习策略的差异依赖之间的相关性。我们观察到泛化与基于模型而非无模型的选择策略之间存在显著的正相关关系。这些结果与基于模型的强化学习的假设是一致的，就像获得性等价一样，依赖于一个更通用的关系记忆系统。
It has recently become widely appreciated that value-based decision making is supported by multiple computational strategies. In particular, animal and human behavior in learning tasks appears to include habitual responses described by prominent model-free reinforcement learning (RL) theories, but also more deliberative or goal-directed actions that can be characterized by a different class of theories, model-based RL. The latter theories evaluate actions by using a representation of the contingencies of the task (as with a learned map of a spatial maze), called an “internal model.” Given the evidence of behavioral and neural dissociations between these approaches, they are often characterized as dissociable learning systems, though they likely interact and share common mechanisms. In many respects, this division parallels a longstanding dissociation in cognitive neuroscience between multiple memory systems, describing, at the broadest level, separate systems for declarative and procedural learning. Procedural learning has notable parallels with model-free RL: both involve learning of habits and both are known to depend on parts of the striatum. Declarative memory, by contrast, supports memory for single events or episodes and depends on the hippocampus. The hippocampus is thought to support declarative memory by encoding temporal and spatial relations among stimuli and thus is often referred to as a relational memory system. Such relational encoding is likely to play an important role in learning an internal model, the representation that is central to model-based RL. Thus, insofar as the memory systems represent more general-purpose cognitive mechanisms that might subserve performance on many sorts of tasks including decision making, these parallels raise the question whether the multiple decision systems are served by multiple memory systems, such that one dissociation is grounded in the other. Here we investigated the relationship between model-based RL and relational memory by comparing individual differences across behavioral tasks designed to measure either capacity. Human subjects performed two tasks, a learning and generalization task (acquired equivalence) which involves relational encoding and depends on the hippocampus; and a sequential RL task that could be solved by either a model-based or model-free strategy. We assessed the correlation between subjects’ use of flexible, relational memory, as measured by generalization in the acquired equivalence task, and their differential reliance on either RL strategy in the decision task. We observed a significant positive relationship between generalization and model-based, but not model-free, choice strategies. These results are consistent with the hypothesis that model-based RL, like acquired equivalence, relies on a more general-purpose relational memory system.
登录
查看更多内容
影响因子：
4.3
作者：
Bornstein AM;Daw ND
通讯作者：
Daw ND
影响因子：
16.2
作者：
Daw ND;Gershman SJ;Seymour B;Dayan P;Dolan RJ
通讯作者：
Dolan RJ
影响因子：
1.4
作者：
DAVIS, H
通讯作者：
DAVIS, H
影响因子：
64.8
作者：
Bunsey, M;Eichenbaum, H
通讯作者：
Eichenbaum, H
影响因子：
2.6
作者：
CORKIN, S
通讯作者：
CORKIN, S