The neural computations supporting hierarchical reinforcement learning

支持分层强化学习的神经计算

• 批准号: 10359201
• 负责人: Anne G.E. Collins
• 金额: $ 38.01万
• 依托单位: UNIVERSITY OF CALIFORNIA BERKELEY
• 依托单位国家: 美国
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-04-01 至 2024-01-31
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/10359201
• 关键词: Address; Anterior; Anxiety; Area; Artificial Intelligence; Attention deficit hyperactivity disorder; Automobile Driving; Basal Ganglia; Behavior; Behavioral; Brain; Clinical; Coffee; Complex; Computer Models; Corpus striatum structure; Curiosities; Data; Decision Making; Diagnosis; Disease; Dopamine; Educational process of instructing; Electroencephalography; Environment; Experimental Models; Functional Magnetic Resonance Imaging; Functional disorder; Goals; Human; Impairment; Individual; Intelligence; Lead; Learning; Life; Light; Mental Depression; Mental disorders; Modeling; Motor; Outcome; Outcomes Research; Performance; Persons; Play; Population; Prefrontal Cortex; Process; Protocols documentation; Psyche structure; Psychiatry; Psychological reinforcement; Rain; Research Domain Criteria; Rewards; Role; Schizophrenia; Sensory; Signal Transduction; Source; Structure; System; Testing; Thinking; Variant; Water; Work; autism spectrum disorder; base; bean; cognitive system; executive function; experimental study; flexibility; insight; learning algorithm; mathematical model; neuromechanism; novel; prevent; relating to nervous system; skills; theories; tool

The neural computations supporting hierarchical reinforcement learning - Project Summary. This project explores how humans learn at multiple hierarchical levels in parallel, and how this supports human intelligence. Human decisions are typically hierarchically structured: we make high-level decisions (making a cup of coffee), which constrain lower level decisions (grinding coffee beans, boiling water, etc.), which themselves constrain simpler and simpler decisions and motor actions. This hierarchy in decisions is paralleled by a hierarchy in our representation of our environment: some sensory signals trigger simple decisions (a red light signals a stop), while other signal a broader, more abstract behavioral change (rain signals a set of adaptations when driving). Thus, complex hierarchical structure underlies the way we respond to our environment in seemingly simple, everyday tasks. This ability is supported by the prefrontal cortex, which represents states and decisions at multiple degrees of hierarchical abstraction. My previous work shows that hierarchical representations support transfer and generalization while learning, an ability that artificial agents still struggle to match human performance in. However, how we learn to form these hierarchical representations is poorly understood, despite how crucial it is for human intelligence. The proposed work will examine how multiple, parallel hierarchical loops between prefrontal cortex and the basal ganglia support reinforcement learning at multiple hierarchical levels in parallel, and how this promotes flexible behavior. To this end, we will address three aims: 1. We will show that the same reinforcement learning computations happen in parallel at multiple levels of abstraction, as hypothesized by our computational model of prefrontal- subcortical networks. 2. We will demonstrate that humans partition learning problems into multiple sequential subgoals so they can learn multiple simple strategies instead of one complex strategy, and that reusing these simple strategies promotes fast exploration and learning. 3. We will show that hierarchical learning does not rely exclusively on rewards, but that novelty signals are crucial for identifying subgoals and learning through curiosity. Across all three aims, we will use behavioral experiments in conjunction with computational modeling to characterize how humans learn hierarchically. In addition, we will use EEG and fMRI to identify the neural computations underlying the cognitive systems inferred from behavior and modeling. This project will provide new insights into the computational mechanisms that give rise to learning, and thus provide a better handle on the sources of learning dysfunction observed in many psychiatric diseases, including schizophrenia, depression, anxiety, ADHD, and OCD. Additionally, it will provide new tools, in the form of experimental protocols and precise computational models, for studying learning across populations and species.

支持分层强化学习的神经计算-项目摘要。 这个项目探讨了人类如何在多个层次上并行学习，以及这如何支持人类的学习。 智能人类的决策通常是分层结构的：我们做出高层次的决策（做出一个 一杯咖啡），这限制了较低级别的决策（研磨咖啡豆、煮沸水等），这 它们本身限制了越来越简单的决定和运动动作。这种决策层次结构是不合理的。 通过我们对环境的表现层次：一些感官信号触发简单的决定（红色 光信号停止），而其他信号更广泛，更抽象的行为变化（雨信号一组 驾驶时的适应）。因此，复杂的等级结构是我们对我们的 在看似简单的日常工作中。这种能力是由前额叶皮层支持的， 表示多级层次抽象的状态和决策。我以前的工作表明， 层次表示支持学习时的迁移和泛化，这是人工智能 仍在努力与人类的表现相匹配。然而，我们如何学会形成这些层次 尽管表征对人类智能至关重要，但人们对它的理解却很少。拟议的工作将 研究前额叶皮层和基底神经节之间的多个平行层次循环是如何支持 并行地在多个层次上进行强化学习，以及这如何促进灵活的行为。到 为此，我们将致力于三个目标：1。我们将证明，同样的强化学习计算 在多个抽象层次上并行发生，正如我们的前额叶计算模型所假设的那样- 皮层下网络2.我们将证明人类将学习问题划分为多个顺序 这样他们就可以学习多个简单的策略，而不是一个复杂的策略， 简单的策略促进快速探索和学习。3.我们将证明，分层学习并不 完全依赖于奖励，但新奇的信号对于识别子目标和学习至关重要。 好奇在所有三个目标中，我们将使用行为实验与计算建模相结合 来描述人类是如何分层次学习的。此外，我们将使用EEG和fMRI来识别神经元 从行为和建模推断出的认知系统的基础计算。本项目将提供 对引起学习的计算机制的新见解，从而更好地处理 在许多精神疾病，包括精神分裂症， 抑郁焦虑多动症和强迫症此外，它还将以实验性的形式提供新的工具， 协议和精确的计算模型，用于研究跨种群和物种的学习。