Human reinforcement learning subdivides structured action spaces by learning effector-specific values.
Human reinforcement learning subdivides structured action spaces by learning effector-specific values.
复制标题
人类强化学习通过学习效应特异性值细分了结构化的作用空间。
DOI：
10.1523/jneurosci.2469-09.2009
复制
发表时间：
2009-10-28
期刊：
影响因子：
--
通讯作者：
Daw ND
中科院分区：
文献类型：
--
作者：
Gershman SJ;Pesaran B;Daw ND
Humans and animals are endowed with a large number of effectors. Although this enables great behavioral flexibility, it presents an equally formidable reinforcement learning problem of discovering which actions are most valuable, due to the high dimensionality of the action space. An unresolved question is how neural systems for reinforcement learning – such as prediction error signals for action valuation associated with dopamine and the striatum – can cope with this “curse of dimensionality.” We propose a reinforcement learning framework that allows for learned action valuations to be decomposed into effector-specific components when appropriate to a task, and test it by studying to what extent human behavior and BOLD activity can exploit such a decomposition in a multieffector choice task. Subjects made simultaneous decisions with their left and right hands and received separate reward feedback for each hand movement. We found that choice behavior was better described by a learning model that decomposed the values of bimanual movements into separate values for each effector, rather than a traditional model that treated the bimanual actions as unitary with a single value. A decomposition of value into effector-specific components was also observed in value-related BOLD signaling, in the form of lateralized biases in striatal correlates of prediction error and anticipatory value correlates in the intraparietal sulcus. These results suggest that the human brain can use decomposed value representations to “divide and conquer” reinforcement learning over high-dimensional action spaces.
