Computional interactions between model-free and model-based decision making in the human brain

适应行为的一个基本原则是，人类和动物寻求最大限度的奖励和最小的惩罚。据推测，大脑采用了两种互补的计算策略来解决这一任务：一种是对以前成功的行为进行无模型重复，另一种是更严格的基于模型的方法，以评估哪些未来的行动将导致最有价值的结果。与这些策略一致的是，决策过程中的神经成像研究揭示了在大脑中进行广泛训练后，与计划和选择有关的明显价值信号的证据。然而，这两个系统是单独行动，还是在学习和选择中相互作用，仍有待观察。此外，控制哪个系统控制行为的因素仍然不清楚。在这项建议中，我们概述了一系列全面的人类研究，以研究无模型选择系统和基于模型的选择系统之间的相互作用，并揭示大脑中与潜在计算过程相对应的信号。具体地说，我们解决了系统之间的信息传输问题，例如通过规划系统或基于模型的教学信号访问学习的缓存值以指示缓存值。然后，我们将研究决定无模型或基于模型的系统是否影响选择以及影响程度的因素和神经机制。特别是，我们将研究相对确定性在两个控制器对行为控制的预测中的假定作用，以及工作记忆性能在两个系统中的作用。最后，我们将通过比较抑郁症受试者和健康对照组的无模型和基于模型的选择绩效和潜在的神经系统，来探索决策在精神病理学中的作用。我们预计，这些发现将极大地增强我们对人脑如何执行决策背后的计算的知识，并可能帮助我们理解和描述精神疾病患者的决策缺陷。

A basic tenet of adaptive behavior is that humans and animals seek to maximize reward and minimize punishment. It has been conjectured that the brain employs two complementary computational strategies to solve this task: a model-free repetition of previously successful behavior, and a more demanding model‐based approach to assess which future actions will lead to the most highly valued outcomes. Consistent with these strategies, neuroimaging studies during decision-making have revealed evidence for distinct value signals pertaining to planning and choice after extensive training in the brain. However, it remains to be seen whether both systems act in isolation or interact in learning and choice. Furthermore, the factors governing which system controls behavior are still unclear. In this proposal we outline a comprehensive series of investigations in humans to study interactions between model-free and model-based choice systems, and to uncover signals in the brain corresponding to the underlying computational processes. Specifically, we address the question of information transfer between systems, such as in accessing learnt cached values by a planning system or a model‐based teaching signal to instruct cached values. We will then investigate the factors and neural mechanisms that determine whether model‐free or model-based systems influence choice and to what degree. Particularly, we will study a putative role of relative certainties in both controllers’ predictions for behavioral control, and the role of working memory performance in both systems. Finally, we will explore the role of decision making in psychopathology by comparing model-free and model‐based choice performance and underlying neural systems between depressed subjects and healthy controls. We expect that the findings will greatly enhance our knowledge about how the human brain implements the computations underlying decision-making and may also help us understand and characterize decision-making deficits in psychiatric patients.