Circuit mechanisms of arbitration between distinct reinforcement learning systems

不同强化学习系统之间的仲裁电路机制

• 批准号: 10608739
• 负责人: Margaret Louise DeMaegd
• 金额: $ 7.41万
• 依托单位: NEW YORK UNIVERSITY
• 依托单位国家: 美国
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-03-01 至 2026-02-28
• 项目状态: 未结题
• 来源: https://reporter.nih.gov/project-details/10608739
• 关键词: Address; Anatomy; Animals; Arbitration; Behavior; Behavior monitoring; Behavioral Model; Brain; Chronic; Computing Methodologies; Confocal Microscopy; Corpus striatum structure; Data; Decision Making; Disease; Dorsal; Electrophysiology (science); Environment; Excitatory Postsynaptic Potentials; Exhibits; Future; Genetic; Goals; Human; Implant; In Vitro; Injections; Interneurons; Knowledge; Lateral; Lead; Learning; Logic; Measures; Medial; Mediating; Methods; Microelectrodes; Modeling; Neurons; Obsessive-Compulsive Disorder; Pattern; Population; Prefrontal Cortex; Process; Psyche structure; Psychological reinforcement; Rattus; Reporting; Structure; Synapses; System; Techniques; Testing; Theoretical Studies; Tracer; Uncertainty; Viral; Visual; Whole-Cell Recordings; Work; behavioral study; cell type; cognitive task; experience; flexibility; free behavior; in vivo; microscopic imaging; neural; neural circuit; novel; novel therapeutics; optogenetics; response; simulation; statistics

PROJECT SUMMARY Animals can exhibit goal-directed behaviors in novel environments, despite limited experience with them. How does the brain make and use inferences about the underlying statistics and generative structure of environments to guide behavior? The field of reinforcement learning refers to this capacity as “model-based” reasoning, meaning that it relies on an internal model of the structure of the world. Critically, this internal model can be used to flexibly estimate the best actions by mental simulation or planning, without direct experience. In contrast, in “model-free” reinforcement learning, an agent chooses the best action based on direct experience, without explicit knowledge of the underlying sequential transition structure of a task or environment. Model-based and model-free mechanisms coexist in the brain and are mediated by distinct circuits, although the neural circuit mechanisms by which the brain arbitrates between these decision systems remains unknown. Theoretical and behavioral studies suggest that human brains use the system that yields value estimates with the lowest uncertainty. The lateral orbitofrontal cortex (lOFC) is a compelling candidate to perform arbitration because while it is implicated in model-based reasoning, for instance by enabling inferences about hidden task states, it lies upstream of the dorsal striatum, which is critical for both model-based and model- free decision making. Intriguingly, we have found that lOFC neurons project exclusively to the dorsolateral striatum (DLS), a region critical for model-free behavior, and not the dorsomedial striatum (DMS), which is critical for model-based behavior. We hypothesize that projection specific neural circuits in lOFC arbitrate between these systems by suppressing the model-free system. I will use state-of-the-art viral, electrophysiological, and computational methods to determine whether DLS-projecting lOFC neurons mediate uncertainty-based arbitration between decision-making systems (Aim 1) and characterize the underlying circuit logic that supports arbitration (Aim 2). By optogenetically tagging DLS-projecting lOFC neurons I will selectively characterize and perturb their activity while monitoring the behavioral strategy rats use in a task with latent structure. To determine how arbitration is instantiated in the dorsal striatum I will optogenetically activate OFC→DLS neurons while recording from different genetic cell types in the striatum, in vivo and in vitro. We predict that OFC→DLS neurons enable model-based behavior by activating inhibitory interneurons to suppress the DLS and the model-free system.

项目摘要 动物可以在新颖环境中表现出目标指导的行为，目的地有限的经验 和他们在一起。大脑如何对基础统计数据进行推断和使用 环境的通用结构以指导行为？增强学习领域是指 以这种能力为“基于模型”的推理，这意味着它依赖于 世界结构。至关重要的是，该内部模型可用于灵活估计最好的 通过精神模拟或计划的行动，没有直接经验。相反，在“无模型”中 加强学习，代理商选择了基于直接经验的最佳行动，而没有 对任务或环境的基本顺序过渡结构的明确知识。 基于模型的基于模型和模型的机制在大脑中共存，并由不同的 电路，尽管神经回路机制在这些机制之间进行了仲裁 决策系统仍然未知。理论和行为研究表明人类 大脑使用以最低不确定性产生的价值估计的系统。后来 Orbitrontal Cortex（LOFC）是执行仲裁的令人信服的候选人 在基于模型的推理中实施，例如通过启用有关隐藏任务的信息 状态，它位于背纹状体的上游，这对于基于模型的模型和模型至关重要 免费决策。有趣的是，我们发现LOFC神经元专门针对 背景纹状体（DLS），一个对无模型行为至关重要的区域，而不是背侧的区域 纹状体（DMS），这对于基于模型的行为至关重要。我们假设该投影 LOFC中的特定神经回路通过抑制无模型来仲裁 系统。 我将使用最先进的病毒，电生理和计算方法 确定是否基于基于不确定性的DLS介绍LOFC神经元之间的仲裁 决策系统（AIM 1）并表征支持的基础电路逻辑 仲裁（目标2）。通过对dls-projection lofc神经元进行吻合标记，我将有选择地 在监视行为策略老鼠在任务中使用的行为策略时表征和扰动他们的活动 具有潜在结构。要确定在背纹状体中如何实例化仲裁 从不同遗传细胞类型记录的同时，将OFC→DLS神经元进行光源激活 纹状体，体内和体外。我们预测OFC→DLS神经元可实现基于模型的 通过激活抑制性神经元抑制DLS和无模型系统来行为。