Task complexity interacts with state-space uncertainty in the arbitration between model-based and model-free learning.
Task complexity interacts with state-space uncertainty in the arbitration between model-based and model-free learning.
复制标题
在基于模型和无模型学习之间的仲裁中，任务复杂性与状态空间不确定性相互作用。
DOI：
10.1101/393983
复制
发表时间：
2020
影响因子：
16.6
通讯作者：
Kim, D. Park
中科院分区：
文献类型：
--
作者：
Kim, D. Park
一个主要的悬而未决的问题涉及大脑如何管理两种不同的强化学习策略之间的控制分配：基于模型的强化学习和无模型的强化学习。虽然有证据表明两个系统预测的可靠性是仲裁过程的一个关键变量，但另一个关键变量仍然相对未被探索：任务复杂性的作用。通过结合新颖的任务设计、计算建模和基于模型的功能磁共振成像分析，我们研究了任务复杂性以及状态空间不确定性在基于模型和无模型强化学习之间仲裁过程中的作用。我们发现的证据表明，任务复杂性与状态空间不确定性一起影响仲裁过程。参与者倾向于增加基于模型的强化学习控制，以应对不断增加的任务复杂性。然而，当不确定性和任务复杂性都很高时，他们诉诸无模型强化学习，这表明这两个变量在仲裁过程中相互作用。计算功能磁共振成像显示，任务复杂性与双侧下前额皮质中两个系统可靠性的神经表征相互作用。这些发现提供了关于下前额叶皮层如何在存在不确定性和复杂性的情况下协商基于模型和无模型强化学习之间的权衡的见解，更一般地说，说明了大脑如何在动态变化的环境中解决不确定性和复杂性。研究结果摘要 - 阐明了状态空间不确定性和复杂性在基于模型和无模型强化学习中的作用。 - 找到了复杂性敏感的前额叶仲裁的行为和神经证据。 - 高任务复杂性引发探索性的基于模型的学习RL。
A major open question concerns how the brain governs the allocation of control between two distinct strategies for learning from reinforcement: model-based and model-free reinforcement learning. While there is evidence to suggest that the reliability of the predictions of the two systems is a key variable responsible for the arbitration process, another key variable has remained relatively unexplored: the role of task complexity. By using a combination of novel task design, computational modeling, and model-based fMRI analysis, we examined the role of task complexity alongside state-space uncertainty in the arbitration process between model-based and model-free RL. We found evidence to suggest that task complexity plays a role in influencing the arbitration process alongside state-space uncertainty. Participants tended to increase model-based RL control in response to increasing task complexity. However, they resorted to model-free RL when both uncertainty and task complexity were high, suggesting that these two variables interact during the arbitration process. Computational fMRI revealed that task complexity interacts with neural representations of the reliability of the two systems in the inferior prefrontal cortex bilaterally. These findings provide insight into how the inferior prefrontal cortex negotiates the trade-off between model-based and model-free RL in the presence of uncertainty and complexity, and more generally, illustrates how the brain resolves uncertainty and complexity in dynamically changing environments.SUMMARY OF FINDINGS- Elucidated the role of state-space uncertainty and complexity in model-based and model-free RL.- Found behavioral and neural evidence for complexity-sensitive prefrontal arbitration.- High task complexity induces explorative model-based RL.