Investigating a Mechanism of Goal-Directed Action

 描述(由申请人提供) 人类和动物如何利用过去的经验来指导未来的决策？强化学习的计算模型为回答这个问题提供了一个有用的框架。虽然一个著名的理论使用了无模型强化学习来描述中脑多巴胺神经元如何在许多经历中增量地计算执行一个动作所获得的奖励的平均水平，但这个理论无法解释动物和人类在了解到该动作的结果的享乐性价值发生变化后灵活地改变他们对该动作的偏好的能力。这种能力被称为目标导向行动，需要预测行动的后果，并评估这些后果如何与目标保持一致。确定目标导向行动背后的神经机制对公共健康至关重要，因为包括强迫症、帕金森氏病和精神分裂症在内的一系列精神和神经疾病都与这种能力和执行这一能力的神经回路缺陷有关。基于模型的强化学习是一种通过预测和评估行动的长期后果来选择行动的框架，它为理解目标导向的行动提供了一个很有前途的理论解释。然而，基于模型的强化学习所规定的计算是如此费力，以至于任何具有有限计算资源的物理系统都必须采取措施来简化它们。这一建议提出了一种生物学上看似合理的机制，通过这种机制，大脑简化了基于模型的强化学习所需的计算，以便执行目标导向的行动。我们的核心假设是，大脑通过存储和重复使用关于行动后果的聚合多步预测来简化基于模型的计算。我们仔细地设计了一个多步骤强化学习任务，对于这个提出的机制，我们产生了可识别的行为，我们提供了初步数据，表明人类表现出这种行为。在目标1中，我们建议使用功能磁共振成像(FMRI)数据的多变量分析来评估决策时的神经活动是否支持这一机制。在目标2中，我们将分析颞叶病变患者的行为，以便将所提出的机制与神经底物因果联系起来。通过拟议的工作，在计算建模和行为分析方面已经相当有经验的主要调查员将获得功能磁共振成像以及神经心理学方法方面的专业知识。此外，这项工作将构成主要调查员毕业论文剩余工作的基础，并将在未来两年内进行。共同赞助者将在整个过程中提供指导，特别是在分析神经成像数据方面。这项工作将在科学会议上公布，并将在完成后发表并向公众提供。

 DESCRIPTION (provided by applicant) How do humans and animals utilize past experience to guide future decisions? Computational models from reinforcement learning have provided a useful framework for answering this question. Whereas a prominent theory has used model-free reinforcement learning to describe how midbrain dopamine neurons incrementally, over many experiences, compute the average of rewards received for performing an action, this theory cannot explain the ability of animals and humans to flexibly change their preference for an action after learning that the hedonic value of its outcome has changed. This ability, referred to as goal-directed action, requires anticipating the consequences of an action and evaluating how those consequences line up with one's goals. Determining the neural mechanisms underlying goal-directed action is of critical importance to public health as a wide range of psychiatric and neurological disorders including Obsessive Compulsive Disorder, Parkinson's disease and Schizophrenia have all been associated with deficits in this ability and the neural circuitry believed to carry it out. Model-based reinforcement learning, a framework for choosing actions by forecasting and evaluating their long-term consequences, offers a promising theoretical account for understanding goal- directed action. However, the computations prescribed by model-based reinforcement learning are so laborious that any physical system with limited computational resources must take steps to simply them. This proposal suggests a biologically plausible mechanism by which the brain simplifies computations required by model-based reinforcement learning in order to perform goal-directed action. Our core hypothesis is that the brain simplifies model-based computations by storing and reusing aggregate multi-step predictions about action consequences. We have carefully designed a multi-step reinforcement-learning task for which this proposed mechanism generates recognizable behavior and we present preliminary data suggesting that humans display this behavior. In Aim 1, we propose using multivariate analysis of functional Magnetic Resonance Imaging (fMRI) data to evaluate whether neural activity at the time of decision-making supports this proposed mechanism. In Aim 2, we will analyze the behavior of temporal lobe lesion patients in order to causally link the proposed mechanism to a neural substrate. Through the proposed work, the Primary Investigator, who is already quite well experienced in computational modeling and behavioral analysis, will acquire expertise in fMRI as well as neuropsychological methods. In addition, this work will form the basis of the remainder of the Primary Investigator's graduate dissertation work and will be carried out over the next 2 years. The co-sponsors will provide guidance throughout the process and particularly with analysis of neuroimaging data. The work will be presented at scientific meetings and will be published and made available to the public when completed.