The interaction of memory and reinforcement learning in value-based choice

基于价值的选择中记忆与强化学习的相互作用

• 批准号: 412917403
• 负责人: Professor Dr. Radoslaw Martin Cichy, since 7/2022
• 金额: --
• 依托单位: Max-Planck-Institut für Bildungsforschung
• 依托单位国家: 德国
• 项目类别: Research Grants
• 财政年份: 2018
• 资助国家: 德国
• 起止时间: 2017-12-31 至 2022-12-31
• 项目状态: 已结题
• 来源: https://gepris.dfg.de/gepris/projekt/412917403?language=en
• 关键词: interaction; memory; reinforcement; learning; value

To make successful decisions in dynamic environments, human decision makers must adaptively change the values they attribute to different choice options. Many value-based decisions rely on long-term memory. Nevertheless, decision making has generally been investigated in isolation from long-term memory. Recent studies suggest that interactions between the hippocampus (HC), ventral striatum (VST), and ventromedial prefrontal cortex (VMPFC) are crucial for memory-guided decisions. Importantly, these brain regions mature at different rates and show substantial individual differences in structure and function. It is currently not known how maturational changes of relevant brain regions from middle childhood to early adulthood as well as individual differences in adulthood alter the interaction between memory and decision-making systems. The first aim of this project is to neurocomputationally dissect the contribution of the HC, VST, and VMPFC, respectively, to memory-based, value-guided decisions. Based on this dissection, the second aim is to develop a mechanistic model of the extent and direction of functional connections among the HC, VST, and VMPFC when making memory-based, value-guided decisions. This model will then be used to formally capture how individual differences in brain structure and function among younger adults contribute to individual differences in decision making. Finally, the third aim of the project is to examine how maturational brain changes from middle childhood to early adulthood modulate memory-valuation interactions. In pursuit of these goals, we will combine a multi-modal, model-based, neuroimaging approach with a neurodevelopmental perspective.In the first work package (WP1), we will neurocomputationally dissect the contribution of the HC, VST, and VMPFC, respectively, in realizing associative memory-guided, value-based decision-making in adults. To this end we will use a model-based fMRI approach, combining fMRI experiments in young adults performing a memory-guided, value-based decision-making task, and modeling behavioral data using reinforcement learning. Next, we will test the extent and direction of the connections between the HC, VST, and VMPFC in realizing associative memory-guided, value-based decision making using dynamic causal modeling (DCM). In the second WP, we will test how individual differences in brain structure (DTI), brain function (fMRI), and behavior (accuracy, RT, participant specific model parameter estimates) are related to each other during value updating and memory-guided decision making. In the third WP, we will adapt the experimental paradigm used in WP1 and WP2 for children aged 8–10 years and adolescents aged 13–15 years. Then we will use model-based fMRI, as well as DTI, to examine the degree to which age-related, differences in structural (DTI) and functional connectivity (fMRI) are associated with age differences in memory-guided, value-based decision making.

为了在动态环境中做出成功的决策，人类决策者必须自适应地改变他们赋予不同选择选项的价值。许多基于价值的决策依赖于长期记忆。然而，决策的研究通常是与长期记忆分开进行的。最近的研究表明，海马体（HC）、腹侧纹状体（VST）和腹内侧前额叶皮层（VMPFC）之间的相互作用对记忆引导的决策至关重要。重要的是，这些大脑区域以不同的速度成熟，并在结构和功能上表现出实质性的个体差异。目前尚不清楚从童年中期到成年早期相关大脑区域的成熟变化以及成年期的个体差异如何改变记忆和决策系统之间的相互作用。该项目的第一个目标是神经计算剖析HC、VST和VMPFC分别对基于记忆的价值导向决策的贡献。在此基础上，第二个目标是建立一个机制模型，说明在做出基于记忆的、价值导向的决策时，HC、VST和VMPFC之间的功能连接的程度和方向。然后，这个模型将用于正式捕捉年轻人大脑结构和功能的个体差异如何导致决策的个体差异。最后，该项目的第三个目标是研究从童年中期到成年早期的成熟大脑变化如何调节记忆评估相互作用。为了实现这些目标，我们将结合多模态、基于模型的神经成像方法和神经发育视角。在第一个工作包（WP1）中，我们将分别从神经计算上剖析HC、VST和VMPFC在实现成人联想记忆引导的、基于价值的决策中的贡献。为此，我们将使用基于模型的功能磁共振成像方法，结合年轻成年人执行记忆引导，基于价值的决策任务的功能磁共振成像实验，并使用强化学习建模行为数据。接下来，我们将使用动态因果模型（DCM）来测试HC、VST和VMPFC之间在实现联想记忆引导、基于价值的决策方面的联系的程度和方向。在第二个WP中，我们将测试在价值更新和记忆引导决策过程中，大脑结构（DTI）、大脑功能（fMRI）和行为（准确性、RT、参与者特定模型参数估计）的个体差异是如何相互关联的。在第三个工作计划中，我们将在8-10岁儿童和13-15岁青少年中采用WP1和WP2中使用的实验范式。然后，我们将使用基于模型的fMRI和DTI来检查年龄相关的结构（DTI）和功能连接（fMRI）差异与记忆导向的基于价值的决策的年龄差异的关联程度。