The role of distributional reinforcement learning in human neurons during impulsive choices

摘要 人工智能和神经科学的最新发展揭示了强化的神经代码 它代表了对一系列可能的未来奖励结果的预测，而不是单一的预期值。 这种分布式强化学习提高了人工代理的性能， 对许多神经精神疾病的直接影响，特别是冲动控制和物质 使用障碍。这项提议旨在利用我们记录大脑神经元活动的经验， 人类神经外科患者，以便将这些记录转化为一个新的研究方向： 人类选择行为的机制。我们将确定分布代码存在于人类 前额叶和内侧颞叶皮层，以及这些代码是如何动态表达的时间作为人类 在气球模拟风险任务（BART）和概率反向学习任务中做出冲动的选择。 这些实验的结果将具有重要的基础科学意义，并将开始解决 人类大脑中的分布式强化学习如何促进冲动选择。 为了开始将这一新的知识领域转化为理解人类决策的基础， 我们将首先在四个大脑区域中建立分布式强化学习的存在，这四个大脑区域包括： 人类决策回路：眶额皮层、前扣带皮层、杏仁核和海马。 具体目标1将测试分布RL的三个基本预测： 这些大脑区域中的每一个都表现出1）奖励预测误差的不对称缩放，2）不同的逆转点， 以及3）预测误差不对称性和反转点跨神经元相关。具体目标2旨在 从上述脑区域中的神经元解码BART奖励预测分布，并确定 BART奖励分布的变化如何与冲动选择的倾向相关。具体 目标3将测试从上述大脑中记录的每个神经元中乐观和悲观的多样性 区域与试验中的估值或贬值相关。 这些目标的实现将成为发现分布式强化学习的重要基础研究成果， 人类的前额叶和内侧颞叶皮层通过揭示神经群体密码 潜在的冲动选择在人类决策电路，这些实验也解决了基本的 冲动选择背后的神经机制这一问题是解决以下重要问题的核心： 当代精神健康包括物质使用障碍和许多其他神经精神障碍。 这些发现将对改善精神病患者的靶向电治疗有很好的启示。 紊乱

ABSTRACT Recent developments in artificial intelligence and neuroscience have revealed neural codes for reinforcement that represent predictions of a range of possible future reward outcomes, rather than a singular expected value. This distributional reinforcement learning has enabled improved performance of artificial agents and has straightforward implications for numerous neuropsychiatric disorders, particularly impulse control and substance use disorders. This proposal aims to leverage our experience recording neuronal activity from the brains of human neurosurgical patients in order to translate these recordings in a novel research direction: to understand the mechanisms of human choice behavior. We will determine where distributional codes exist in the human prefrontal and mesial temporal cortices, and how those codes are expressed dynamically in time as humans make impulsive choices during the Balloon Analog Risk Task (BART) and a probabilistic reversal learning task. The results of these experiments will have both important basic scientific implications and will begin to address how distributional reinforcement learning in the human brain contributes to impulsive choices. In order to begin translating this new area of knowledge to understand the underpinnings of human decisions, we will first establish the presence of distributional reinforcement learning in four brain areas that comprise a human decision-making circuit: Orbitofrontal Cortex, Anterior Cingulate Cortex, Amygdala, and Hippocampus. Specific Aim 1 will test the three essential predictions of distributional RL: whether populations of neurons in each of these brain areas exhibit 1) asymmetric scaling of reward prediction errors, 2) diverse reversal points, and 3) that prediction error asymmetries and reversal points correlate across neurons. Specific Aim 2 seeks to decode BART reward prediction distributions from neurons in the aforementioned brain areas and determine how changes in BART reward distributions correlate with the propensity to make impulsive choices. Specific Aim 3 will test how diversity in optimism and pessimism in each neuron recorded from the aforementioned brain areas correlates with valuation or devaluation across trials. The completion of these aims will constitute important basic research findings in discovering distributional RL in the human prefrontal and mesial temporal cortices. By uncovering neural population codes that underlie potentially impulsive choices in human decision-making circuits, these experiments also address fundamental neural mechanisms underlying impulsive choices. This issue is central to addressing important problems for contemporary mental health including substance use disorder and a many other neuropsychiatric disorders. These findings will have readily translatable implications for improving targeted electrical therapies for psychiatric disorders.