Reinforcement Learning in Multidimensional Environments Relies on Attention Mechanisms
Reinforcement Learning in Multidimensional Environments Relies on Attention Mechanisms
复制标题
DOI：
10.1523/jneurosci.2978-14.2015
复制
发表时间：
2015-05-27
影响因子：
5.3
通讯作者：
Wilson, Robert C.
中科院分区：
文献类型：
--
作者：
Niv, Yael;Daniel, Reka;Wilson, Robert C.
近年来，强化学习计算领域的思想彻底改变了大脑学习的研究，最著名的是提供了关于多巴胺如何影响基底神经节学习的新的、精确的理论。然而，强化学习算法因无法很好地扩展到多维环境而臭名昭著，而这正是现实世界学习所需的。我们假设大脑自然地将现实世界问题的维度减少到仅与预测奖励相关的维度，并进行了一项实验来评估通过什么算法和什么神经机制在人类中实现这种“表征学习”过程。我们的结果表明，由顶内沟、楔前叶和背外侧前额叶皮层组成的双边注意力控制网络参与选择与手头任务相关的维度，通过反复试验有效地更新任务表征。这样，皮质注意力机制与基底神经节的学习相互作用，解决强化学习中的“维数灾难”。
In recent years, ideas from the computational field of reinforcement learning have revolutionized the study of learning in the brain, famously providing new, precise theories of how dopamine affects learning in the basal ganglia. However, reinforcement learning algorithms are notorious for not scaling well to multidimensional environments, as is required for real-world learning. We hypothesized that the brain naturally reduces the dimensionality of real-world problems to only those dimensions that are relevant to predicting reward, and conducted an experiment to assess by what algorithms and with what neural mechanisms this "representation learning" process is realized in humans. Our results suggest that a bilateral attentional control network comprising the intraparietal sulcus, precuneus, and dorsolateral prefrontal cortex is involved in selecting what dimensions are relevant to the task at hand, effectively updating the task representation through trial and error. In this way, cortical attention mechanisms interact with learning in the basal ganglia to solve the "curse of dimensionality" in reinforcement learning.