The neural computations supporting hierarchical reinforcement learning

支持分层强化学习的神经计算--项目总结。 这个项目探索了人类如何并行地在多个层次上学习，以及这如何支持人类 智慧。人类的决策通常是有层次结构的：我们做出高级别的决策(做出 咖啡)，这限制了较低级别的决策(研磨咖啡豆、沸水等)，这些决定 它们本身制约着越来越简单的决定和运动动作。决策中的这种等级是平行的 根据我们对环境的表示中的层次结构：一些感觉信号会触发简单的决定(红色 光表示停止)，而其他表示更广泛、更抽象的行为变化(雨表示一组 驾驶时的适应)。因此，复杂的等级结构是我们回应我们的 在看似简单的日常任务中创造良好的环境。这种能力由前额叶皮质支持，前额叶皮质 表示多个层次抽象程度的状态和决策。我之前的工作表明， 层次表示支持边学习边迁移和泛化，这是人工智能体 仍在努力与人类的表现相媲美。然而，我们如何学习形成这些等级 尽管陈述对人类智力有多么重要，但人们对它的理解很少。拟议的工作将 研究前额叶皮质和基底神经节之间的多个平行层次环路如何支持 多个层级并行的强化学习，以及这如何促进灵活行为。至 为此，我们将解决三个目标：1.我们将证明相同的强化学习计算 在多个抽象层次上并行发生，正如我们的前额叶计算模型所假设的那样- 皮质下网络。2.我们将演示人类将学习问题划分为多个顺序 子目标，这样他们就可以学习多个简单的策略，而不是一个复杂的策略，并重复使用这些策略 简单的策略有助于快速探索和学习。3.我们将证明分层学习不会 完全依赖于奖励，但新奇的信号对于确定子目标和学习至关重要 好奇心。在所有三个目标中，我们将结合使用行为实验和计算建模 来刻画人类如何层次分明地学习。此外，我们将使用脑电和功能磁共振来识别神经 从行为和建模中推断出的认知系统的潜在计算。该项目将提供 对促进学习的计算机制的新见解，从而提供了更好的处理 在包括精神分裂症在内的许多精神疾病中观察到的学习障碍的来源， 抑郁、焦虑、多动症和强迫症。此外，它将提供新的工具，以试验性的形式 协议和精确的计算模型，用于研究跨种群和物种的学习。

The neural computations supporting hierarchical reinforcement learning - Project Summary. This project explores how humans learn at multiple hierarchical levels in parallel, and how this supports human intelligence. Human decisions are typically hierarchically structured: we make high-level decisions (making a cup of coffee), which constrain lower level decisions (grinding coffee beans, boiling water, etc.), which themselves constrain simpler and simpler decisions and motor actions. This hierarchy in decisions is paralleled by a hierarchy in our representation of our environment: some sensory signals trigger simple decisions (a red light signals a stop), while other signal a broader, more abstract behavioral change (rain signals a set of adaptations when driving). Thus, complex hierarchical structure underlies the way we respond to our environment in seemingly simple, everyday tasks. This ability is supported by the prefrontal cortex, which represents states and decisions at multiple degrees of hierarchical abstraction. My previous work shows that hierarchical representations support transfer and generalization while learning, an ability that artificial agents still struggle to match human performance in. However, how we learn to form these hierarchical representations is poorly understood, despite how crucial it is for human intelligence. The proposed work will examine how multiple, parallel hierarchical loops between prefrontal cortex and the basal ganglia support reinforcement learning at multiple hierarchical levels in parallel, and how this promotes flexible behavior. To this end, we will address three aims: 1. We will show that the same reinforcement learning computations happen in parallel at multiple levels of abstraction, as hypothesized by our computational model of prefrontal- subcortical networks. 2. We will demonstrate that humans partition learning problems into multiple sequential subgoals so they can learn multiple simple strategies instead of one complex strategy, and that reusing these simple strategies promotes fast exploration and learning. 3. We will show that hierarchical learning does not rely exclusively on rewards, but that novelty signals are crucial for identifying subgoals and learning through curiosity. Across all three aims, we will use behavioral experiments in conjunction with computational modeling to characterize how humans learn hierarchically. In addition, we will use EEG and fMRI to identify the neural computations underlying the cognitive systems inferred from behavior and modeling. This project will provide new insights into the computational mechanisms that give rise to learning, and thus provide a better handle on the sources of learning dysfunction observed in many psychiatric diseases, including schizophrenia, depression, anxiety, ADHD, and OCD. Additionally, it will provide new tools, in the form of experimental protocols and precise computational models, for studying learning across populations and species.