Reinforcement learning and action sequencing in subcortical and cortical circuits

诸如动作选择和动作排序等行为需要动态神经活动模式的塑造 通过学习。了解这种学习是如何发生的是具有挑战性的，因为涉及多个 由于这些行为涉及多个时间尺度，从颗粒水平来看， 即时肢体控制到目标驱动计划的认知水平。现代实验， 能够记录行为动物的大量神经元，并且在某些情况下能够同时记录 在多个大脑区域和整个任务的学习过程中，为解决这些问题提供了一条前进的道路。 挑战我的研究的总体目标是促进对这些数据的综合和理解， 实验通过构建与给定行为相关的大脑回路模型，解决 这些回路中的神经活动与行为以及它如何通过学习随着时间的推移而形成有关。 在最近的工作中，我通过三条相关的路线， research.首先，我对与时间相关的行为及其实现的神经计算进行了建模 在基底神经节。其次，我从数学上推导出了生物学上合理的学习规则， 递归神经网络中时间相关任务的监督学习。最后，我做了一个理论实验， 协作，其中使用递归神经网络建模与脑机协作 在猴子中进行接口实验，以解决初级运动中神经表征的结构 皮层在未来的工作中，我将建立在这一经验，以解决动态神经活动模式是如何 为了在短期和长期的时间尺度上产生复杂的行为而学习。 开始解决这个问题的一种方法是强化学习理论，它提供了丰富的 这是一个强大的框架，用于解决如何执行行动以最大化未来回报。 鉴于它们在实施强化学习中的既定作用，基底神经节形成了起点 我提议的研究项目首先，我的目标是通过构建基底神经节功能的经典模型， 并通过数学分析解决计算挑战性任务的模型， 用我的实验合作者的新数据来验证结果（目标1）。在这项工作的基础上， 根据我之前训练递归神经网络来模拟运动任务的经验，我接下来将考虑学习 运动皮层以及它如何补充基底神经节的学习（目标2），再次将模型与新的实验模型进行比较。 数据最后，我将构建模型的丘脑皮质基底神经节电路， 了解整个回路的神经表征，并利用机器的最新进展， 学习通过这种方式，我将解决哺乳动物的大脑如何实现层次强化 学习整合短期和长期的行为（目标3）。综合考虑，这项研究将推动 了解神经活动如何促进复杂行为中的动作选择和排序。

Behaviors such as action selection and action sequencing require the shaping of dynamical neural activity patterns through learning. Understanding how such learning occurs is challenging due to the involvement of multiple brain areas and due to the fact that such behaviors involve multiple timescales, from the granular level of moment-to-moment limb control to the cognitive level of goal-driven planning. Modern experiments, which are able to record from large numbers of neurons in behaving animals, and in some cases to do so simultaneously in multiple brain areas and throughout the learning of a task, are providing a path forward for addressing these challenges. The overall goal of my research is to facilitate the synthesis and understanding of data from such experiments by constructing models of the brain circuits relevant for a given behavior, addressing how the neural activity in these circuits relates to behavior and how it is shaped over time through learning. In recent work, I have developed expertise in learned dynamics in neural circuits through three related lines of research. First, I have modeled the neural computations underlying timing-related behavior and its implementation in the basal ganglia. Second, I have mathematically derived biologically plausible learning rules to underlie supervised learning of time-dependent tasks in recurrent neural networks. Finally, I have worked on a theory-experiment collaboration in which modeling with recurrent neural networks was used in tandem with brain-machine interface experiments in monkeys to address the structure of neural representations within primary motor cortex. In future work, I will build on this experience to address how dynamical neural activity patterns are learned in order to produce complex behaviors over both short and long timescales. One way to begin addressing this question is with the theory of reinforcement learning, which provides a rich and powerful framework for addressing how actions should be performed in order to maximize future rewards. Given their established role in implementing reinforcement learning, the basal ganglia form the starting point for my proposed research program. I first aim to revise the classical model of basal ganglia function by constructing and mathematically analyzing models that solve computationally challenging tasks and by comparing the results with new data from my experimental collaborators (Aim 1). Building on this work, and making use of my prior experience training recurrent neural networks to model motor tasks, I will next consider learning in motor cortex and how it complements learning in basal ganglia (Aim 2), again comparing models with new experimental data. Finally, I will construct models of the thalamocortico-basal ganglia circuit by incorporating knowledge about the neural representations throughout this circuit and by leveraging recent advances in machine learning. In this way I will address how the mammalian brain implements hierarchical reinforcement learning to integrate behaviors over short and long timescales (Aim 3). Taken together, this research will advance understanding of how neural activity facilitates action selection and sequencing in complex behaviors.