Reinforcement learning and action sequencing in subcortical and cortical circuits

动作选择和动作排序等行为需要形成动态的神经活动模式 通过学习。理解这种学习是如何发生的是具有挑战性的，因为 大脑区域，并由于这种行为涉及多个时间尺度的事实，从颗粒水平 从瞬间肢体控制到认知层面的目标驱动规划。现代实验，它们是 能够从行为动物的大量神经元中进行记录，在某些情况下还可以同时这样做 在多个大脑区域和在学习任务的整个过程中，都为解决这些问题提供了一条前进的道路 挑战。我研究的总体目标是促进从这些数据中综合和理解 通过构建与给定行为相关的大脑回路模型来进行实验，解决如何 这些回路中的神经活动与行为以及它是如何通过学习随着时间的推移而形成的有关。 在最近的工作中，我通过三个相关的系列发展了神经电路中学习动力学的专业知识 研究。首先，我对与时间相关的行为及其实现背后的神经计算进行了建模 在基底节。其次，我有从数学上推导出的生物学上看似合理的学习规则作为基础 递归神经网络中时间相关任务的有监督学习。最后，我做了一个理论实验 在协作中，使用递归神经网络建模与脑-机器相结合 猕猴初级运动中神经表征结构的接口实验 大脑皮层。在未来的工作中，我将在这一经验的基础上解决动态神经活动模式是如何 学习是为了在短时间和长时间范围内产生复杂的行为。 解决这个问题的一种方法是使用强化学习理论，它提供了丰富的 以及强大的框架，用于解决应该如何执行行动，以最大化未来的回报。 鉴于它们在实施强化学习中的既定作用，基底节是起点 我提议的研究项目。首先，我的目标是修正经典的基底节功能模型 对解决具有计算挑战性的任务的模型进行数学分析，并通过比较 使用我的实验合作者的新数据得出的结果(目标1)。在此工作的基础上，并利用 我以前训练递归神经网络来模拟运动任务的经验，接下来我会考虑学习 运动皮质及其如何补充基底神经节的学习(目标2)，再次将模型与新的实验进行比较 数据。最后，我将构建丘脑皮质-基底节回路的模型 关于整个电路中神经表示的知识，并利用机器的最新进展 学习。通过这种方式，我将阐述哺乳动物的大脑是如何实现等级强化的 学会在短时间和长时间内整合行为(目标3)。综上所述，这项研究将推动 了解神经活动如何促进复杂行为中的动作选择和排序。

Behaviors such as action selection and action sequencing require the shaping of dynamical neural activity patterns through learning. Understanding how such learning occurs is challenging due to the involvement of multiple brain areas and due to the fact that such behaviors involve multiple timescales, from the granular level of moment-to-moment limb control to the cognitive level of goal-driven planning. Modern experiments, which are able to record from large numbers of neurons in behaving animals, and in some cases to do so simultaneously in multiple brain areas and throughout the learning of a task, are providing a path forward for addressing these challenges. The overall goal of my research is to facilitate the synthesis and understanding of data from such experiments by constructing models of the brain circuits relevant for a given behavior, addressing how the neural activity in these circuits relates to behavior and how it is shaped over time through learning. In recent work, I have developed expertise in learned dynamics in neural circuits through three related lines of research. First, I have modeled the neural computations underlying timing-related behavior and its implementation in the basal ganglia. Second, I have mathematically derived biologically plausible learning rules to underlie supervised learning of time-dependent tasks in recurrent neural networks. Finally, I have worked on a theory-experiment collaboration in which modeling with recurrent neural networks was used in tandem with brain-machine interface experiments in monkeys to address the structure of neural representations within primary motor cortex. In future work, I will build on this experience to address how dynamical neural activity patterns are learned in order to produce complex behaviors over both short and long timescales. One way to begin addressing this question is with the theory of reinforcement learning, which provides a rich and powerful framework for addressing how actions should be performed in order to maximize future rewards. Given their established role in implementing reinforcement learning, the basal ganglia form the starting point for my proposed research program. I first aim to revise the classical model of basal ganglia function by constructing and mathematically analyzing models that solve computationally challenging tasks and by comparing the results with new data from my experimental collaborators (Aim 1). Building on this work, and making use of my prior experience training recurrent neural networks to model motor tasks, I will next consider learning in motor cortex and how it complements learning in basal ganglia (Aim 2), again comparing models with new experimental data. Finally, I will construct models of the thalamocortico-basal ganglia circuit by incorporating knowledge about the neural representations throughout this circuit and by leveraging recent advances in machine learning. In this way I will address how the mammalian brain implements hierarchical reinforcement learning to integrate behaviors over short and long timescales (Aim 3). Taken together, this research will advance understanding of how neural activity facilitates action selection and sequencing in complex behaviors.