The interaction of memory and reinforcement learning in value-based choice

为了在动态环境中做出成功的决策，人类决策者必须自适应地改变他们赋予不同选择选项的价值。许多基于价值的决定依赖于长期记忆。然而，人们通常是从长期记忆的角度来研究决策的。最近的研究表明，海马体(HC)、腹侧纹状体(VST)和腹内侧额叶皮质(VMPFC)之间的相互作用对记忆引导的决策至关重要。重要的是，这些大脑区域的成熟速度不同，在结构和功能上表现出显著的个体差异。目前尚不清楚从童年中期到成年早期相关脑区的成熟变化以及成年期的个体差异如何改变记忆和决策系统之间的相互作用。这个项目的第一个目标是从神经计算的角度分别剖析HC、VST和VMPFC对基于记忆、价值导向的决策的贡献。基于这一解剖，第二个目标是建立一个机制模型，来描述在做出基于记忆、价值导向的决策时，HC、VST和VMPFC之间的功能联系的程度和方向。然后，这个模型将被用来正式捕捉年轻人大脑结构和功能的个体差异如何导致决策过程中的个体差异。最后，该项目的第三个目标是研究从儿童期中期到成年期早期大脑的成熟变化如何调节记忆-价值的交互作用。为了追求这些目标，我们将结合多模式、基于模型的神经成像方法和神经发育的观点。在第一个工作包(WP1)中，我们将分别从神经计算的角度剖析HC、VST和VMPFC在实现成人联想记忆引导的基于价值的决策中的贡献。为此，我们将使用基于模型的功能磁共振方法，结合执行记忆引导、基于价值的决策任务的年轻人的功能磁共振实验，并使用强化学习对行为数据进行建模。接下来，我们将测试在使用动态因果模型(DCM)实现联想记忆引导的基于价值的决策时，HC、VST和VMPFC之间的联系的程度和方向。在第二个WP中，我们将测试在价值更新和记忆引导的决策过程中，脑结构(DTI)、脑功能(FMRI)和行为(准确性、RT、参与者特定模型参数估计)的个体差异是如何相互关联的。在第三个WP中，我们将调整WP1和WP2中使用的实验范式，以适用于8-10岁的儿童和13-15岁的青少年。然后，我们将使用基于模型的功能磁共振成像以及DTI，来检查在记忆引导的、基于价值的决策中，年龄相关、结构差异(DTI)和功能连接差异(FMRI)与年龄差异的程度。

To make successful decisions in dynamic environments, human decision makers must adaptively change the values they attribute to different choice options. Many value-based decisions rely on long-term memory. Nevertheless, decision making has generally been investigated in isolation from long-term memory. Recent studies suggest that interactions between the hippocampus (HC), ventral striatum (VST), and ventromedial prefrontal cortex (VMPFC) are crucial for memory-guided decisions. Importantly, these brain regions mature at different rates and show substantial individual differences in structure and function. It is currently not known how maturational changes of relevant brain regions from middle childhood to early adulthood as well as individual differences in adulthood alter the interaction between memory and decision-making systems. The first aim of this project is to neurocomputationally dissect the contribution of the HC, VST, and VMPFC, respectively, to memory-based, value-guided decisions. Based on this dissection, the second aim is to develop a mechanistic model of the extent and direction of functional connections among the HC, VST, and VMPFC when making memory-based, value-guided decisions. This model will then be used to formally capture how individual differences in brain structure and function among younger adults contribute to individual differences in decision making. Finally, the third aim of the project is to examine how maturational brain changes from middle childhood to early adulthood modulate memory-valuation interactions. In pursuit of these goals, we will combine a multi-modal, model-based, neuroimaging approach with a neurodevelopmental perspective.In the first work package (WP1), we will neurocomputationally dissect the contribution of the HC, VST, and VMPFC, respectively, in realizing associative memory-guided, value-based decision-making in adults. To this end we will use a model-based fMRI approach, combining fMRI experiments in young adults performing a memory-guided, value-based decision-making task, and modeling behavioral data using reinforcement learning. Next, we will test the extent and direction of the connections between the HC, VST, and VMPFC in realizing associative memory-guided, value-based decision making using dynamic causal modeling (DCM). In the second WP, we will test how individual differences in brain structure (DTI), brain function (fMRI), and behavior (accuracy, RT, participant specific model parameter estimates) are related to each other during value updating and memory-guided decision making. In the third WP, we will adapt the experimental paradigm used in WP1 and WP2 for children aged 8–10 years and adolescents aged 13–15 years. Then we will use model-based fMRI, as well as DTI, to examine the degree to which age-related, differences in structural (DTI) and functional connectivity (fMRI) are associated with age differences in memory-guided, value-based decision making.