Flexibility to contingency changes distinguishes habitual and goal-directed strategies in humans
Flexibility to contingency changes distinguishes habitual and goal-directed strategies in humans
复制标题
DOI：
10.1371/journal.pcbi.1005753
复制
发表时间：
2017-09-01
影响因子：
4.3
通讯作者：
Keramati, Mehdi
中科院分区：
文献类型：
--
作者：
Lee, Julie J.;Keramati, Mehdi
现实世界中的决策提出了要求灵活而迅速的行为的挑战，这种平衡的特点是在较慢的、前瞻性目标导向的基于模型（MB）策略和快速的、回顾性的习惯性无模型（MF）策略之间进行权衡。理论预测，对奖励值和过渡偶然性变化的灵活性可以决定这两个系统在强化学习中的相对影响，但很少有研究操纵后者。因此，我们开发了一种新的两级权变任务，其中状态之间的过渡权变每隔几次试验就会发生变化；MB和MF控制预测了这些偶然性变化后的不同反应，从而可以推断出它们的相对影响。此外，我们操纵了偶然性变化的速率，以确定偶然性变化的波动性是否会在MB和MF策略之间的转移中发挥作用。我们发现人类受试者在任务中采用了混合的MB/MF策略，证实了MB和MF系统在强化学习中的平行贡献。此外，受试者并没有保持在一个MB/MF行为水平上，而是在前两个区块中表现出向更多MB行为的转变，这不是由于偶然性变化的速度，而是由于训练的程度。我们证明了对偶然性变化的灵活性可以区分MB和MF策略，人类受试者使用混合策略，在街区中转向更多的MB行为，从而对应更高的回报。
Decision-making in the real world presents the challenge of requiring flexible yet prompt behavior, a balance that has been characterized in terms of a trade-off between a slower, prospective goal-directed model-based (MB) strategy and a fast, retrospective habitual model-free (MF) strategy. Theory predicts that flexibility to changes in both reward values and transition contingencies can determine the relative influence of the two systems in reinforcement learning, but few studies have manipulated the latter. Therefore, we developed a novel two-level contingency change task in which transition contingencies between states change every few trials; MB and MF control predict different responses following these contingency changes, allowing their relative influence to be inferred. Additionally, we manipulated the rate of contingency changes in order to determine whether contingency change volatility would play a role in shifting subjects between a MB and MF strategy. We found that human subjects employed a hybrid MB/MF strategy on the task, corroborating the parallel contribution of MB and MF systems in reinforcement learning. Further, subjects did not remain at one level of MB/MF behaviour but rather displayed a shift towards more MB behavior over the first two blocks that was not attributable to the rate of contingency changes but rather to the extent of training. We demonstrate that flexibility to contingency changes can distinguish MB and MF strategies, with human subjects utilizing a hybrid strategy that shifts towards more MB behavior over blocks, consequently corresponding to a higher payoff.