The neural computations supporting hierarchical reinforcement learning

支持分层强化学习的神经计算

• 批准号: 10359201
• 负责人: Anne G.E. Collins
• 金额: $ 38.01万
• 依托单位: UNIVERSITY OF CALIFORNIA BERKELEY
• 依托单位国家: 美国
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-04-01 至 2024-01-31
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/10359201
• 关键词: Address; Anterior; Anxiety; Area; Artificial Intelligence; Attention deficit hyperactivity disorder; Automobile Driving; Basal Ganglia; Behavior; Behavioral; Brain; Clinical; Coffee; Complex; Computer Models; Corpus striatum structure; Curiosities; Data; Decision Making; Diagnosis; Disease; Dopamine; Educational process of instructing; Electroencephalography; Environment; Experimental Models; Functional Magnetic Resonance Imaging; Functional disorder; Goals; Human; Impairment; Individual; Intelligence; Lead; Learning; Life; Light; Mental Depression; Mental disorders; Modeling; Motor; Outcome; Outcomes Research; Performance; Persons; Play; Population; Prefrontal Cortex; Process; Protocols documentation; Psyche structure; Psychiatry; Psychological reinforcement; Rain; Research Domain Criteria; Rewards; Role; Schizophrenia; Sensory; Signal Transduction; Source; Structure; System; Testing; Thinking; Variant; Water; Work; autism spectrum disorder; base; bean; cognitive system; executive function; experimental study; flexibility; insight; learning algorithm; mathematical model; neuromechanism; novel; prevent; relating to nervous system; skills; theories; tool

The neural computations supporting hierarchical reinforcement learning - Project Summary. This project explores how humans learn at multiple hierarchical levels in parallel, and how this supports human intelligence. Human decisions are typically hierarchically structured: we make high-level decisions (making a cup of coffee), which constrain lower level decisions (grinding coffee beans, boiling water, etc.), which themselves constrain simpler and simpler decisions and motor actions. This hierarchy in decisions is paralleled by a hierarchy in our representation of our environment: some sensory signals trigger simple decisions (a red light signals a stop), while other signal a broader, more abstract behavioral change (rain signals a set of adaptations when driving). Thus, complex hierarchical structure underlies the way we respond to our environment in seemingly simple, everyday tasks. This ability is supported by the prefrontal cortex, which represents states and decisions at multiple degrees of hierarchical abstraction. My previous work shows that hierarchical representations support transfer and generalization while learning, an ability that artificial agents still struggle to match human performance in. However, how we learn to form these hierarchical representations is poorly understood, despite how crucial it is for human intelligence. The proposed work will examine how multiple, parallel hierarchical loops between prefrontal cortex and the basal ganglia support reinforcement learning at multiple hierarchical levels in parallel, and how this promotes flexible behavior. To this end, we will address three aims: 1. We will show that the same reinforcement learning computations happen in parallel at multiple levels of abstraction, as hypothesized by our computational model of prefrontal- subcortical networks. 2. We will demonstrate that humans partition learning problems into multiple sequential subgoals so they can learn multiple simple strategies instead of one complex strategy, and that reusing these simple strategies promotes fast exploration and learning. 3. We will show that hierarchical learning does not rely exclusively on rewards, but that novelty signals are crucial for identifying subgoals and learning through curiosity. Across all three aims, we will use behavioral experiments in conjunction with computational modeling to characterize how humans learn hierarchically. In addition, we will use EEG and fMRI to identify the neural computations underlying the cognitive systems inferred from behavior and modeling. This project will provide new insights into the computational mechanisms that give rise to learning, and thus provide a better handle on the sources of learning dysfunction observed in many psychiatric diseases, including schizophrenia, depression, anxiety, ADHD, and OCD. Additionally, it will provide new tools, in the form of experimental protocols and precise computational models, for studying learning across populations and species.

支持分层增强学习的神经计算 - 项目摘要。 该项目探讨了人类如何在多个层次结构层面上学习，这如何支持人类 智力。人类的决定通常是层次结构化的：我们做出高级决定（做出 咖啡杯），这限制了较低级别的决策（磨咖啡豆，开水等） 自己限制了更简单，更简单的决策和运动动作。决策中的层次结构是平行的 通过我们对环境的表示的层次结构：一些感官信号触发了简单的决定（红色 灯标志着停止），而其他信号是更广泛的，更抽象的行为改变（雨水信号一组 开车时进行改编）。因此，复杂的层次结构是我们对我们的响应方式的基础 看似简单的日常任务中的环境。这种能力得到了前额叶皮层的支持， 在多个程度的分层抽象中表示状态和决策。我以前的工作表明 层次表示在学习时支持转移和概括，这是人造代理的能力 仍然很难与人类的表现相匹配。但是，我们如何学会形成这些等级制度 尽管对人类的智力是多么重要，但对代表性的理解很少。拟议的工作将 检查前额叶皮层和基底神经节支撑之间的多个平行分层环 并行多个分层的增强学习，以及这如何促进灵活的行为。到 这一目的，我们将解决三个目标：1。我们将证明相同的强化学习计算 我们的前额叶计算模型在多个级别的抽象上并行进行 皮层网络。 2。我们将证明人类将学习问题分为多个顺序 子目标使他们可以学习多种简单策略，而不是一种复杂的策略，然后再将这些策略重复 简单的策略促进了快速的探索和学习。 3。我们将证明分层学习不是 仅依靠奖励，但是新颖的信号对于识别子目标和通过学习至关重要 好奇心。在这三个目标中，我们将使用行为实验与计算建模结合 表征人类如何在层次上学习。此外，我们将使用脑电图和fMRI识别神经 根据行为和建模推断的认知系统的计算。这个项目将提供 对产生学习的计算机制的新见解，从而更好地解决了 在包括精神分裂症在内的许多精神疾病中观察到的学习功能障碍的来源， 抑郁，焦虑，多动症和强迫症。此外，它将以实验的形式提供新工具 方案和精确的计算模型，用于研究跨种群和物种的学习。