The role of distributional reinforcement learning in human neurons during impulsive choices

摘要 人工智能和神经科学的最新发展揭示了强化的神经密码 这代表了对一系列未来可能的奖励结果的预测，而不是一个单一的期望值。 这种分布式强化学习提高了人工代理的性能，并 对许多神经精神障碍的直接影响，特别是冲动控制和物质 使用障碍。这项提议旨在利用我们记录大脑中神经元活动的经验 人类神经外科患者为了将这些录音翻译成一个新的研究方向：理解 人类选择行为的机制。我们将确定人类的分配代码存在于何处 前额叶和内侧颞叶皮质，以及这些编码如何作为人类在时间上动态表达 在气球模拟风险任务(BART)和概率反转学习任务中做出冲动选择。 这些实验的结果将具有重要的基础科学意义，并将开始解决 人脑中的分布式强化学习如何有助于冲动的选择。 为了开始翻译这一新的知识领域，以理解人类决策的基础， 我们将首先在四个大脑区域中建立分布式强化学习的存在，这四个区域包括 人类决策回路：眶前叶皮质、前扣带回、杏仁核和海马体。 特定目标1将测试分布RL的三个基本预测：神经元群体在 这些大脑区域中的每一个都表现出1)奖励预测误差的不对称缩放，2)不同的反转点， 3)预测误差的不对称性和反转点在神经元之间存在关联。具体目标2旨在 解码上述脑区神经元的BART奖赏预测分布，并确定 BART奖励分布的变化如何与冲动选择的倾向相关。特定的 目标3将测试从前述大脑记录的每个神经元中乐观和悲观的多样性 在不同的试验中，区域与估值或贬值相关。 这些目标的完成将成为发现分布RL的重要基础研究成果 人类的前额叶和内侧的颞叶皮质。通过揭示神经种群编码的基础 在人类决策回路中潜在的冲动选择，这些实验也解决了基本问题 冲动选择背后的神经机制。这个问题是解决以下重要问题的核心 当代精神健康包括物质使用障碍和许多其他神经精神障碍。 这些发现将对改善精神疾病的靶向电疗具有很好的可译性。 精神错乱。

ABSTRACT Recent developments in artificial intelligence and neuroscience have revealed neural codes for reinforcement that represent predictions of a range of possible future reward outcomes, rather than a singular expected value. This distributional reinforcement learning has enabled improved performance of artificial agents and has straightforward implications for numerous neuropsychiatric disorders, particularly impulse control and substance use disorders. This proposal aims to leverage our experience recording neuronal activity from the brains of human neurosurgical patients in order to translate these recordings in a novel research direction: to understand the mechanisms of human choice behavior. We will determine where distributional codes exist in the human prefrontal and mesial temporal cortices, and how those codes are expressed dynamically in time as humans make impulsive choices during the Balloon Analog Risk Task (BART) and a probabilistic reversal learning task. The results of these experiments will have both important basic scientific implications and will begin to address how distributional reinforcement learning in the human brain contributes to impulsive choices. In order to begin translating this new area of knowledge to understand the underpinnings of human decisions, we will first establish the presence of distributional reinforcement learning in four brain areas that comprise a human decision-making circuit: Orbitofrontal Cortex, Anterior Cingulate Cortex, Amygdala, and Hippocampus. Specific Aim 1 will test the three essential predictions of distributional RL: whether populations of neurons in each of these brain areas exhibit 1) asymmetric scaling of reward prediction errors, 2) diverse reversal points, and 3) that prediction error asymmetries and reversal points correlate across neurons. Specific Aim 2 seeks to decode BART reward prediction distributions from neurons in the aforementioned brain areas and determine how changes in BART reward distributions correlate with the propensity to make impulsive choices. Specific Aim 3 will test how diversity in optimism and pessimism in each neuron recorded from the aforementioned brain areas correlates with valuation or devaluation across trials. The completion of these aims will constitute important basic research findings in discovering distributional RL in the human prefrontal and mesial temporal cortices. By uncovering neural population codes that underlie potentially impulsive choices in human decision-making circuits, these experiments also address fundamental neural mechanisms underlying impulsive choices. This issue is central to addressing important problems for contemporary mental health including substance use disorder and a many other neuropsychiatric disorders. These findings will have readily translatable implications for improving targeted electrical therapies for psychiatric disorders.