Ensemble neural dynamics in the medial prefrontal cortex underlying cognitive flexibility and reinforcement learning

摘要 人们认为前额叶皮质在认知灵活性方面发挥着至关重要的作用，部分原因是它更新了一个人的认知灵活性 对外部世界的期望以及基于反馈的候选行为的可能后果 从过去的行动中获得的。这种认知形式的缺陷发生在多种精神疾病中， 前额叶皮层也有牵连尽管有很多研究，前额叶神经回路的机制 有助于灵活的决策和认知策略的转换仍不清楚。我们将研究这些 问题使用强化学习理论，它指定了选择未来行动的最佳策略 考虑到受试者过去采取的行动和获得的奖励的历史。首先，我们将收集最大的一组多- 神经元记录，然后使用强化学习理论来分析 数据并推导出前额叶皮质存储和更新其内部信念的电路机制 关于外部世界和未来行动的可能结果。 过去对行为动物的研究发现，平均而言， 编码与认知策略和行动选择相关的信息，但由于数据有限， 可以确定前额叶回路如何在多个过程中维持和更新这些信息。 决定、行动和结果。为了收集足够的数据并创建更好的前额叶回路模型，我们 将使用微型显微镜，使我们能够监测活跃小鼠的大型神经系统。我们的目标是： (1)开发并验证一个实验范例，用于成像数百个 前额叶细胞在两种不同的空间导航策略之间灵活切换。我们的试点数据 显示老鼠可以很好地完成任务，前额叶活动对策略转换至关重要， 皮层包含的细胞的动态似乎信号估计的最佳策略。我们将验证小鼠 可以遵循真正的导航策略，而不仅仅是记住产生回报的空间路径。我们还将 确认前额叶细胞在多日实验中保持健康并具有正常的活动模式。 (2)使用强化学习理论来分析我们的大型数据集，并创建神经电路模型， 前额叶皮层储存并更新其信念，以指导未来的行动。利用我们首先要创造的理论 小鼠行为的行为模型。然后，我们将应用数据的监督和非监督方法 分析以评估前额叶神经系统是否编码与任务相关的抽象变量，如信念 和价值使用我们的行为者模型和神经动力学分析，我们将训练递归神经网络， 网络模型来解决策略转换任务。由此产生的强化学习电路模型将 然后产生可测试的预测，即当我们修改任务时，小鼠和前额叶细胞应该如何表现。 总的来说，我们的研究将解决有关前额叶功能的关键未回答的问题，并寻求实现 对前额叶回路如何有助于灵活决策的机械理解。

Abstract The prefrontal cortex is thought to play a crucial role in cognitive flexibility, in part by updating a person's expectations about the external world and the likely consequences of candidate actions based on the feedback gained from past actions. Deficits in this form of cognition occur in multiple psychiatric conditions in which prefrontal cortex is implicated. Despite much research, the mechanisms by which prefrontal neural circuits contribute to flexible decision-making and switches in cognitive strategy remain unclear. We will examine these issues using reinforcement learning theory, which specifies the optimal strategies for selecting future actions given a subject's past history of actions taken and rewards received. We will first gather the largest set of multi- neuronal recordings ever taken in prefrontal cortex, and then use reinforcement learning theory to analyze the data and deduce the circuit mechanisms by which the prefrontal cortex stores and updates its internal beliefs about the external world and the likely results of future actions. Past studies in behaving animals have found evidence for individual prefrontal cells that, on average, encode information related to cognitive strategy and action selection, but with limited data it has not been possible to identify how prefrontal circuits maintain and update this information over the course of multiple decisions, actions and outcomes. To collect sufficient data and create better models of prefrontal circuits, we will use a miniature microscope enabling us to monitor large neural ensembles in active mice. Our goals are to: (1) Develop and validate an experimental paradigm for imaging the concurrent dynamics of hundreds of prefrontal cells in mice flexibly switching between two different strategies of spatial navigation. Our pilot data show mice can perform the task well, that prefrontal activity is crucial for strategy-switching, and that prefrontal cortex contains cells whose dynamics appear to signal estimates of the optimal strategy. We will verify mice can follow bona fide navigation strategies and not just memorize spatial paths that yield reward. We will also confirm the prefrontal cells stay healthy and have normal activity patterns throughout the multi-day experiment. (2) Use reinforcement learning theory to analyze our large datasets and create neural circuit models of how prefrontal cortex stores and updates its beliefs to guide future actions. Using the theory we will first create observer-actor models of mouse behavior. We will then apply supervised and unsupervised methods of data analysis to assess whether prefrontal neural ensembles encode task-related, abstract variables such as belief and value. Using our observer-actor models and analyses of neural dynamics, we will train recurrent neural network models to solve the strategy-switching task. The resulting circuit models of reinforcement learning will then yield testable predictions about how the mice and prefrontal cells should behave when we modify the task. Overall, our study will address key unanswered questions about prefrontal function and seeks to attain a mechanistic understanding of how prefrontal circuits contribute to flexible decision-making.