CRCNS: Collaborative Research: Neural Correlates of Hierarchical Reinforcement Learning

点击翻译按钮获取中文摘要

Research on human behavior has long emphasized its hierarchical structure: Simple actions group together into subtask sequences, and these in turn cohere to bring about higher-level goals. This hierarchical structure is critical to humans' unique ability to tackle complex, large-scale tasks, since it allows such tasks to be decomposed or broken down into more manageable parts. While some progress has been made toward understanding the origins and mechanisms of hierarchical behavior, key questions remain: How are task-subtask-action hierarchies initially assembled through learning? How does learning operate within such hierarchies, allowing adaptive hierarchical behavior to take shape? How do the relevant learning and action-selection processes play out in neural hardware? To pursue these questions, the present proposal will leverage ideas emerging from the computational framework of Hierarchical Reinforcement Learning (HRL). HRL builds on a highly successful machine-learning paradigm known as reinforcement learning (RL), extending it to include task-subtask-action hierarchies. Recent neuroscience and behavioral research has suggested that standard RL mechanisms may be directly relevant to reward-based learning in humans and animals. The present proposal hypothesizes that the mechanisms introduced in computational HRL may be similarly relevant, providing insight into the cognitive and neural underpinnings of hierarchical behavior. The project brings together two computational cognitive neuroscientists and a computer scientist with expertise in machine learning. The proposed research, which includes both computational modeling and human functional neuroimaging and behavioral studies, pursues a set of hypotheses drawn directly from HRL research. A first set of hypotheses relates to the question of how complex tasks are decomposed into manageable subtasks. Here, fMRI and computational work will leverage the idea, drawn from HRL research, that useful decompositions "carve" tasks at points identifiable through graph-theoretic measures of centrality. A second set of hypotheses relates to the question of how learning occurs within hierarchies. Here, fMRI and modeling work will pursue the idea that hierarchical learning may be driven by reward prediction errors akin to those arising within the HRL framework. The project as a whole aims to construct a biologically constrained neural network model, translating computational HRL into an account of how the brain supports hierarchically structured behavior.