Neural Prediction Errors Reveal a Risk-Sensitive Reinforcement-Learning Process in the Human Brain
Neural Prediction Errors Reveal a Risk-Sensitive Reinforcement-Learning Process in the Human Brain
复制标题
DOI：
10.1523/jneurosci.5498-10.2012
复制
发表时间：
2012-01-11
影响因子：
5.3
通讯作者：
O'Doherty, John P.
中科院分区：
文献类型：
--
作者：
Niv, Yael;Edlund, Jeffrey A.;O'Doherty, John P.
人类和动物对风险或行为结果的变化都非常敏感，尽管这是一种特殊的敏感。当明确提供有关风险的信息时，经济、心理和神经方面的问题得到了很好的研究。然而，我们通常必须通过试验和错误从经验中学习结果。这种强化学习的传统模型专注于学习线索的平均奖励值，而忽略了方差等高阶矩。我们使用fMRI来测试人类强化学习的神经相关性是否对经历的风险敏感。我们的分析集中在解剖学上划定的先验感兴趣的区域，在那里，血氧水平依赖（BOLD）信号被认为与来自强化学习的量相关。我们首先提供了无偏的证据，表明这些区域中的原始BOLD信号与奖励预测误差密切相关。然后，我们从这个信号中获得的经验值的线索，预测奖励相等的意思，但不同的方差，并表明这些值确实是调制经验的风险。此外，我们通过神经测量的风险期权经验评估波动与行为风险规避波动之间存在密切的神经测量-心理测量耦合。这表明，风险敏感性是人类学习不可或缺的一部分，它阐明了选择的经济模型、情感学习的神经科学模型以及潜在神经机制的运作。
Humans and animals are exquisitely, though idiosyncratically, sensitive to risk or variance in the outcomes of their actions. Economic, psychological, and neural aspects of this are well studied when information about risk is provided explicitly. However, we must normally learn about outcomes from experience, through trial and error. Traditional models of such reinforcement learning focus on learning about the mean reward value of cues and ignore higher order moments such as variance. We used fMRI to test whether the neural correlates of human reinforcement learning are sensitive to experienced risk. Our analysis focused on anatomically delineated regions of a priori interest in the nucleus accumbens, where blood oxygenation level-dependent (BOLD) signals have been suggested as correlating with quantities derived from reinforcement learning. We first provide unbiased evidence that the raw BOLD signal in these regions corresponds closely to a reward prediction error. We then derive from this signal the learned values of cues that predict rewards of equal mean but different variance and show that these values are indeed modulated by experienced risk. Moreover, a close neurometric-psychometric coupling exists between the fluctuations of the experience-based evaluations of risky options that we measured neurally and the fluctuations in behavioral risk aversion. This suggests that risk sensitivity is integral to human learning, illuminating economic models of choice, neuroscientific models of affective learning, and the workings of the underlying neural mechanisms.