Distributional reinforcement learning in the brain.

大脑中的分布式强化学习。

• 批准号: 9978224
• 负责人: Jan Drugowitsch
• 金额: $ 178.45万
• 依托单位: HARVARD UNIVERSITY
• 依托单位国家: 美国
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-04-15 至 2023-05-31
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/9978224
• 关键词: Algorithms; Anhedonia; Animals; Artificial Intelligence; Back; Brain; Characteristics; Code; Data Analyses; Development; Dopamine; Eating Disorders; Environment; Event; Feedback; Future; Goals; Investigation; Learning; Mental Depression; Mental disorders; Midbrain structure; Modeling; Modification; Monitor; Mus; Neurobiology; Neurons; Parkinson Disease; Pathologic; Plant Roots; Population; Predictive Value; Probability; Psychological reinforcement; Rewards; Scheme; Schizophrenia; Signal Transduction; Stimulus; Structure; Testing; Ventral Striatum; Ventral Tegmental Area; Work; addiction; artificial neural network; classical conditioning; deep learning; design; dopamine system; dopaminergic neuron; expectation; experimental study; improved; in silico; learning algorithm; network architecture; neuromechanism; optimism; optogenetics; response; simulation; theories

Project Summary The field of artificial intelligence (AI) has recently made remarkable advances that resulted in new and improved algorithms and network architectures that proved efficient empirically in silico. These advances raise new questions in neurobiology: are these new algorithms used in the brain? The present study focuses on a new algorithm developed in the field of reinforcement learning (RL), called distributional RL, which outperforms other state-of-the-art RL algorithms and is regarded as a major advancement in RL. In environments in which rewards are probabilistic with respect to its occurrence and size, traditional RL algorithms have focused on learning to predict a single quantity, the average over all potential rewards. Distributional RL, by contrast, learns to predict the entire distribution over rewards (or values) by employing multiple value predictors that together encode all possible levels of future reward concurrently. Remarkably, theoretical work has shown that a class of distributional RL, called ‘quantile distributional RL’, can arise out of a simple modification of traditional RL that introduces structured variability in dopamine reward prediction error (RPE) signals. This project set out to test the hypothesis that the brain utilizes distributional RL to predict future rewards. Aim 1 will explore the characteristics of distributional RL theoretically and make predictions that allow for testing distributional RL in the brain. Theoretical investigations and simulations will be used to determine how value representations in distributional RL differ from pre-existing population coding schemes for representing probability distributions (probabilistic population codes, distributed distributional codes, etc.). Aim 2 will examine the activity of neurons that are thought to signal RPEs and reward expectation and test various predictions of distributional RL. Specifically, the activity of dopamine neurons in the ventral tegmental area and neurons in the ventral striatum and orbitofrontal cortex will be compared to key predictions of distributional RL. Aim 3 will use optogenetic manipulation to causally demonstrate the relationship between RPE signals and distributional codes.

项目总结