Frontocortical Signaling Signatures in Flexible Reinforcement Learning

各种神经精神疾病导致无法产生奖励环境的准确模型， 无法使用这些模型来指导灵活的行为，通常表现为反向学习受损。 前扣带皮层（ACC）和眶额皮层（OFC）是重要的额皮质区域， 灵活的强化学习，并且理论上可以在并行过程的层次结构中工作，以获得奖励- 基于选择。在OFC中，存在对较低级别属性的优先级编码，如感官的奖励预测值 线索，特定奖励的适口性，以及与行为相关的当前刺激-奖励映射。在行政协调会， 这些变量被认为是多路复用的，用于更高级别的奖励预测误差（RPE）计算， 预测的置信度/不确定性，用于监控性能和更新行为策略 必要时（特别是积极反馈后的总体试验策略，即，WinStay）。这些计算 可能取决于尖峰从OFC到ACC的传播。然而，人们仍然不太清楚， 奖赏学习是由眶额皮层和前额叶皮层之间的相互作用介导的。在这里，我们将研究这个问题 使用不确定性下适应性学习的鲁棒动物模型：基于刺激的概率逆转 学习（PRL）。在自由行为的大鼠中，我们将使用体内单光子钙成像和 电生理学，化学遗传学和奖励传递的闭环神经控制，以研究OFC和 ACC调节PRL。使用我们最近开发的新技术在线解码钙活动 我们将使用一种新的策略，根据ACC和OFC的神经活动来调节奖励传递， 灵活的奖励学习是否依赖于这些额皮质区域的精确神经表征。到 迄今为止，我们已经：在体内和转导的皮质切片中证明了有效的DREADDs操作;设计 并测试了定制的电极阵列，以在这些区域进行长期的体内电生理记录 同时;和成像的合奏活动时间锁定的行为，这已被证明是稳定的多个 会议，理想的学习学习。利用这些技术进步，并利用这一能力作为平台，我们 我建议确定灵活强化中编码变量的精确皮质-皮质机制 学习有两个目的。总的来说，这些实验将：1）揭示新的信号特征， 皮层区域及其各自在灵活强化学习中的作用，2）加速突破性 实验，因为他们将在闭环执行：控制反向学习实时使用解码 神经预期，以及3）这些信号最终将在精神病理学的动物模型中进行比较 因为他们在逆向学习中失败了。这些新颖和非常规的方法使 R21机制的理想建议的工作。

Various neuropsychiatric conditions lead to failures in generating accurate models of the reward environment or inabilities in using those models to guide flexible behavior, very often manifesting as impaired reversal learning. The anterior cingulate cortex (ACC) and the orbitofrontal cortex (OFC) are frontocortical regions important for flexible reinforcement learning, and have been theorized to work in a hierarchy of parallel processes for reward- based choice. In OFC, there is priority encoding of lower-level attributes like reward-predictive value of sensory cues, the palatability of specific rewards, and the current stimulus-reward mappings relevant to behavior. In ACC, these variables are thought to be multiplexed for higher-level computations of reward prediction error (RPE) and confidence/uncertainty of predictions, which are used to monitor performance and update behavioral strategies when necessary (particularly overall trial strategy following positive feedback, i.e., WinStay). These computations may depend upon propagation of spikes from OFC to ACC. However, it remains poorly understood how flexible reward learning is mediated by interactions between OFC and ACC. Here we will investigate this question using a robust animal model of adaptive learning under uncertainty: stimulus-based probabilistic reversal learning (PRL). In freely behaving rats, we will use a combination of in vivo 1-photon calcium imaging and electrophysiology, chemogenetics, and closed-loop neural control of reward delivery to examine how OFC and ACC regulate PRL. Using new technology that we have recently developed for online decoding of calcium activity we will use a novel strategy of regulating reward delivery based upon neural activity in ACC and OFC to test whether flexible reward learning depends upon accurate neural representations in these frontocortical areas. To date, we have: demonstrated effective DREADDs manipulation in vivo and in transduced cortical slices; designed and tested custom electrode arrays to perform chronic in vivo electrophysiological recordings in these areas simultaneously; and imaged ensemble activity time-locked to behavior, which has proven stable over multiple sessions, ideal to study learning. Leveraging these technical advances and using this capacity as a platform, we propose to identify the precise cortico-cortical mechanisms of encoding variables in flexible reinforcement learning across two Aims. Collectively, these experiments will: 1) shed new light on the signaling signatures of cortical regions and their respective roles in flexible reinforcement learning, 2) accelerate groundbreaking experiments as they would be performed in closed-loop: control of reversal learning in real-time using decoded neural expectation, and 3) these signals would eventually be compared in animal models of psychopathology because of their known failures in reversal learning. These novel and unconventional approaches make the R21 mechanism ideal for the proposed work.