CRCNS: Collaborative Research: Neural Correlates of Hierarchical Reinforcement Learning

长期以来，人类行为的研究一直强调其层次结构：简单的行动组合成子任务序列，这些序列反过来又相互联系，从而实现更高层次的目标。 这种层次结构对于人类处理复杂的大规模任务的独特能力至关重要，因为它允许将这些任务分解或分解为更易于管理的部分。虽然在理解层级行为的起源和机制方面取得了一些进展，但关键问题仍然存在：任务-子任务-动作层级最初是如何通过学习组装起来的？ 学习是如何在这样的层级中运作的，从而使适应性层级行为得以形成？ 相关的学习和动作选择过程如何在神经硬件中发挥作用？ 为了解决这些问题，本提案将利用分层强化学习（HRL）计算框架中出现的想法。HRL建立在一个非常成功的机器学习范式之上，称为强化学习（RL），将其扩展到包括任务-子任务-动作层次结构。最近的神经科学和行为研究表明，标准RL机制可能与人类和动物的奖励学习直接相关。目前的建议假设，在计算HRL中引入的机制可能是类似的相关，提供洞察层次行为的认知和神经基础。该项目汇集了两名计算认知神经科学家和一名具有机器学习专业知识的计算机科学家。 拟议的研究，其中包括计算建模和人类功能神经成像和行为研究，追求一套假设直接从HRL研究。 第一组假设涉及如何将复杂任务分解为可管理的子任务的问题。在这里，功能磁共振成像和计算工作将利用的想法，从HRL的研究，即有用的分解“雕刻”的点，通过图论的中心性措施可识别的任务。 第二组假设涉及学习如何在层级中发生的问题。 在这里，功能磁共振成像和建模工作将追求这样的想法，即层次学习可能是由奖励预测错误驱动的，类似于HRL框架中出现的错误。 整个项目的目标是构建一个生物约束的神经网络模型，将计算HRL转换为大脑如何支持分层结构行为的解释。

Research on human behavior has long emphasized its hierarchical structure: Simple actions group together into subtask sequences, and these in turn cohere to bring about higher-level goals. This hierarchical structure is critical to humans' unique ability to tackle complex, large-scale tasks, since it allows such tasks to be decomposed or broken down into more manageable parts. While some progress has been made toward understanding the origins and mechanisms of hierarchical behavior, key questions remain: How are task-subtask-action hierarchies initially assembled through learning? How does learning operate within such hierarchies, allowing adaptive hierarchical behavior to take shape? How do the relevant learning and action-selection processes play out in neural hardware? To pursue these questions, the present proposal will leverage ideas emerging from the computational framework of Hierarchical Reinforcement Learning (HRL). HRL builds on a highly successful machine-learning paradigm known as reinforcement learning (RL), extending it to include task-subtask-action hierarchies. Recent neuroscience and behavioral research has suggested that standard RL mechanisms may be directly relevant to reward-based learning in humans and animals. The present proposal hypothesizes that the mechanisms introduced in computational HRL may be similarly relevant, providing insight into the cognitive and neural underpinnings of hierarchical behavior. The project brings together two computational cognitive neuroscientists and a computer scientist with expertise in machine learning. The proposed research, which includes both computational modeling and human functional neuroimaging and behavioral studies, pursues a set of hypotheses drawn directly from HRL research. A first set of hypotheses relates to the question of how complex tasks are decomposed into manageable subtasks. Here, fMRI and computational work will leverage the idea, drawn from HRL research, that useful decompositions "carve" tasks at points identifiable through graph-theoretic measures of centrality. A second set of hypotheses relates to the question of how learning occurs within hierarchies. Here, fMRI and modeling work will pursue the idea that hierarchical learning may be driven by reward prediction errors akin to those arising within the HRL framework. The project as a whole aims to construct a biologically constrained neural network model, translating computational HRL into an account of how the brain supports hierarchically structured behavior.