Distributional Reinforcement Learning in the Brain

大脑中的分布式强化学习

• 批准号: 10709775
• 负责人: Jan Drugowitsch
• 金额: $ 58.12万
• 依托单位: HARVARD UNIVERSITY
• 依托单位国家: 美国
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-04-15 至 2025-05-31
• 项目状态: 未结题
• 来源: https://reporter.nih.gov/project-details/10709775
• 关键词: Distributional; Reinforcement; Learning; Brain

Project Summary The field of artificial intelligence (AI) has recently made remarkable advances that resulted in new and improved algorithms and network architectures that proved efficient empirically in silico. These advances raise new questions in neurobiology: are these new algorithms used in the brain? The present study focuses on a new algorithm developed in the field of reinforcement learning (RL), called distributional RL, which outperforms other state-of-the-art RL algorithms and is regarded as a major advancement in RL. In environments in which rewards are probabilistic with respect to its occurrence and size, traditional RL algorithms have focused on learning to predict a single quantity, the average over all potential rewards. Distributional RL, by contrast, learns to predict the entire distribution over rewards (or values) by employing multiple value predictors that together encode all possible levels of future reward concurrently. Remarkably, theoretical work has shown that a class of distributional RL, called ‘quantile distributional RL’, can arise out of a simple modification of traditional RL that introduces structured variability in dopamine reward prediction error (RPE) signals. This project set out to test the hypothesis that the brain utilizes distributional RL to predict future rewards. Aim 1 will explore the characteristics of distributional RL theoretically and make predictions that allow for testing distributional RL in the brain. Theoretical investigations and simulations will be used to determine how value representations in distributional RL differ from pre-existing population coding schemes for representing probability distributions (probabilistic population codes, distributed distributional codes, etc.). Aim 2 will examine the activity of neurons that are thought to signal RPEs and reward expectation and test various predictions of distributional RL. Specifically, the activity of dopamine neurons in the ventral tegmental area and neurons in the ventral striatum and orbitofrontal cortex will be compared to key predictions of distributional RL. Aim 3 will use optogenetic manipulation to causally demonstrate the relationship between RPE signals and distributional codes.

项目摘要 人工智能领域（AI）最近取得了显着的进步，导致了新的和 改进的算法和网络体系结构在经验上有效地证明了有效的硅。这些 进步提出了神经生物学中的新问题：这些新算法是否在大脑中使用？这 目前的研究重点是在增强学习领域（RL）中开发的一种新算法，称为 分销RL，其表现优于其他最先进的RL算法，被认为是主要的 RL的进步。在奖励相对于其发生的概率的环境中 和大小，传统的RL算法重点是学习预测单个数量，平均水平 所有潜在的奖励。相比之下，分销RL学会了预测整个分布 奖励（或值）通过使用多个值预测指标，这些预测指标共同编码所有可能的级别 值得注意的是，理论工作表明一类分布RL， 可以通过简单的传统RL修改来引起称为“分数分布RL” 引入多巴胺奖励预测误差（RPE）信号中的结构性变异性。 该项目着手测试大脑利用分布RL预测未来的假设 奖励。 AIM 1将探讨分布RL理论的特征并做出预测 这允许测试大脑中的分布RL。理论研究和模拟将是 用于确定分布RL中的价值表示与预先存在的总体不同 用于表示概率分布的编码方案（概率人口代码，分布 分销代码等）。 AIM 2将检查被认为可以发出RPE的神经元的活性 并奖励期望并测试分布RL的各种预测。具体而言，活动的活动 腹侧侧距区域的多巴胺神经元，腹侧纹状体和轨道额的神经元中的神经元 将皮质与分布RL的关键预测进行比较。 AIM 3将使用光遗传学 操纵以有效的方式证明了RPE信号与分布代码之间的关系。