Cortical and hippocampal correlates of deliberation during model-based decisions for rewards in humans.
Cortical and hippocampal correlates of deliberation during model-based decisions for rewards in humans.
复制标题
DOI：
10.1371/journal.pcbi.1003387
复制
发表时间：
2013
影响因子：
4.3
通讯作者：
Daw ND
中科院分区：
文献类型：
--
作者：
Bornstein AM;Daw ND
我们如何利用过去的记忆来指导我们以前从未做过的决定？尽管大量的工作描述了大脑如何学习重复奖励动作，但决策也可能受到不直接涉及奖励的刺激或事件之间的关联的影响，例如使用认知地图规划路线或使用预测的反击棋步时，这些关联在新选项中做出决定时至关重要。此过程称为基于模型的决策。虽然对可能支持基于模型的决策的环境关系的学习进行了深入研究，并且已经单独推断出此类信息会影响决策，但几乎没有证据表明获得此类关联并推动选择的整个周期。特别令人感兴趣的是，决策是否直接由以更普遍的关系学习为特征的相同助记系统支持，或者依赖于其他专门的表示。在这里，基于我们之前的工作，隔离了顺序预测学习背后的双重表征，我们直接证明了由海马记忆系统和相邻皮质结构编码的一种这样的表征支持目标导向的决策。使用交错的学习和决策任务，我们直接监控预测学习，并追踪其对奖励决策的影响。我们使用计算模型拟合来定量比较多个行为和功能磁共振成像观测值背后的学习过程。在这两项任务中，定量一致的学习过程解释了反应时间、选择以及与期望和惊喜相关的神经活动。在学习过程中参与预测刺激的海马和腹侧流区域的参与程度也与决策的难度成正比。这些结果支持海马记忆系统学习到的预测关联在选择形成过程中被回忆起来的作用。我们总是在学习周围世界的规律：事物在哪里，以及我们可以按照什么顺序找到它们。如果我们以后想要使用这些意外事件的知识来做出决策，那么我们可以依赖这些知识。然而，对于如何利用习得的突发事件进行决策的神经生物学机制，目前还没有达成共识。这些决定与简单的习惯不同，我们采取的行动过去曾给我们带来回报。计算强化学习方法可以很好地描述习惯性决策的神经机制，但通常不应用于依赖于习得的偶然事件的“基于模型”的决策。在本文中，我们应用强化学习来研究基于模型的决策。我们对参与者进行了一系列反应时间任务的测试，其中包括不断变化的连续意外事件以及依赖于这些意外事件的选择探针。将计算模型与反应时间相拟合，我们发现两组预测驱动简单的响应行为，只有其中一组用于做出选择。使用功能磁共振成像，我们观察了海马和腹侧皮质区域的学习和决策相关活动，这些活动在计算上与用于做出选择的学习到的意外事件相关。这些结果表明海马皮质网络在基于模型的奖励决策中发挥着关键作用。
How do we use our memories of the past to guide decisions we've never had to make before? Although extensive work describes how the brain learns to repeat rewarded actions, decisions can also be influenced by associations between stimuli or events not directly involving reward — such as when planning routes using a cognitive map or chess moves using predicted countermoves — and these sorts of associations are critical when deciding among novel options. This process is known as model-based decision making. While the learning of environmental relations that might support model-based decisions is well studied, and separately this sort of information has been inferred to impact decisions, there is little evidence concerning the full cycle by which such associations are acquired and drive choices. Of particular interest is whether decisions are directly supported by the same mnemonic systems characterized for relational learning more generally, or instead rely on other, specialized representations. Here, building on our previous work, which isolated dual representations underlying sequential predictive learning, we directly demonstrate that one such representation, encoded by the hippocampal memory system and adjacent cortical structures, supports goal-directed decisions. Using interleaved learning and decision tasks, we monitor predictive learning directly and also trace its influence on decisions for reward. We quantitatively compare the learning processes underlying multiple behavioral and fMRI observables using computational model fits. Across both tasks, a quantitatively consistent learning process explains reaction times, choices, and both expectation- and surprise-related neural activity. The same hippocampal and ventral stream regions engaged in anticipating stimuli during learning are also engaged in proportion to the difficulty of decisions. These results support a role for predictive associations learned by the hippocampal memory system to be recalled during choice formation. We are always learning regularities in the world around us: where things are, and in what order we might find them. Our knowledge of these contingencies can be relied upon if we later want to use them to make decisions. However, there is little agreement about the neurobiological mechanism by which learned contingencies are deployed for decision making. These are different kinds of decisions than simple habits, in which we take actions that have in the past given us reward. Neural mechanisms of habitual decisions are well-described by computational reinforcement learning approaches, but have not often been applied to ‘model-based’ decisions that depend on learned contingencies. In this article, we apply reinforcement learning to investigate model-based decisions. We tested participants on a serial reaction time task with changing sequential contingencies, and choice probes that depend on these contingencies. Fitting computational models to reaction times, we show that two sets of predictions drive simple response behavior, only one of which is used to make choices. Using fMRI, we observed learning and decision-related activity in hippocampal and ventral cortical areas that is computationally linked to the learned contingencies used to make choices. These results suggest a critical role for a hippocampal-cortical network in model-based decisions for reward.
登录
查看更多内容
影响因子：
64.8
作者：
Daw, Nathaniel D.;O'Doherty, John P.;Dayan, Peter;Seymour, Ben;Dolan, Raymond J.
通讯作者：
Dolan, Raymond J.
影响因子：
16.2
作者：
Daw ND;Gershman SJ;Seymour B;Dayan P;Dolan RJ
通讯作者：
Dolan RJ
影响因子：
5.7
作者：
Buchel, C;Wise, RJS;Friston, KJ
通讯作者：
Friston, KJ
影响因子：
9.2
作者：
Bestmann, Sven;Harrison, Lee M.;Rothwell, John C.
通讯作者：
Rothwell, John C.
影响因子：
64.8
作者：
Bunsey, M;Eichenbaum, H
通讯作者：
Eichenbaum, H