Model-based credit assignment

摘要 大脑如何形成，调整和使用预测模型，这些模型指定了大脑中刺激之间的因果关系。 环境、我们的选择及其结果是心理学和神经科学中的一个基本问题。一 在识别理论上形成和更新的神经计算方面已经取得了很大进展 预测模型这项研究在计算精神病学的兴起中发挥了核心作用， 与临床疾病的相关性部分受到相对简单的学习/选择范例的限制 只捕捉了现实世界学习结构复杂性的一个狭窄子集。 为了在复杂的世界中做出合理的预测，大脑需要将好的和坏的归因于 结果与最可能的原因相对应，这一问题被称为“信用分配”。人们对 在结构化的现实世界环境中，结果如何归因于其最可能的原因。大多数现实世界 学习发生在复杂的和结构化的环境中，例如分层系统（例如季节性事件， 社会等级、上下文规则等）。最近的证据表明，人类可以利用对 环境的因果结构将结果归因于最可能的原因（我称之为“基于模型的 信贷分配）"，而不是简单地归因于他们最近经历的刺激和选择， （我称之为“无模型”信用分配），正如标准模型所提出的那样。 本建议的目的是开发基于模型的信用分配的第一个神经模型。 我们假设，当一个强化的结果被体验到关联时， 对结果的看法换句话说，“一起点火/一起布线”的可塑性机制可以将一个选择 对于结果，选择表示和结果表示必须同时处于活动状态。 时间，即使因果选择或事件可能已经发生了一段时间之前。 为了检验这个和其他的预测，我们将整合学习和决策的数学描述 使用“代表性”分析方法，允许对信息进行推断， 代表大脑区域，应用于功能磁共振成像和头皮脑电图数据。功能磁共振成像将揭示神经学习信号 在学习过程中更新可能原因的神经表征，而EEG将揭示学习的时间。 假设复职这些实验将为应用所获得的见解来研究如何 在普遍存在的等级结构环境中，大脑将结果归因于更抽象的“潜在”原因 在真实的世界里因此，拟议的项目将把这一研究战略的一般方案推向学习 更好地反映许多现实世界学习/选择情况的复杂性和结构的任务， 包括典型和非典型的个体。

Abstract How the brain forms, tunes, and uses predictive models that specify the causal links between stimuli in the environment, our choices, and their outcomes is a fundamental question in Psychology and Neuroscience. A great deal of progress has been made identifying the neural computations theorized to form and update predictive models. This research has played a central role in the rise of computational psychiatry, but its relevance to clinical disorders has been limited in part by the use of relatively simple learning/choice paradigms that capture only a narrow subset of the structural complexity of real-world learning. In order to make sound predictions in a complex world, the brain needs to attribute good and bad outcomes to their most likely causes, a problem known as “credit assignment”. There is little understanding of how outcomes are attributed to their most likely causes in structured real-world environments. Most real-world learning occurs in complex and structured environments, such as hierarchical systems (e.g. seasonal events, social hierarchies, contextual rules, etc.). Recent evidence suggests that humans can use an understanding of the environment’s causal structure to attribute outcomes to their most likely causes (which I call “model-based credit assignment)”, rather than simply attributing them to the most recently experienced stimuli and choices that were made (which I call “model-free” credit assignment), as standard models have proposed. The purpose of the present proposal is to develop the first neural model of model-based credit assignment. We hypothesize that the brain reinstates the cause when a reinforcement outcome is experienced to associate with the outcome. In other words, so that “fire-together/wire-together” plasticity mechanisms can link a choice with an outcome, the choice representation and the outcome representation must both be active at the same time even though the causal choice or event may have actually occurred some time beforehand. To test this and other predictions, we will integrate mathematical descriptions of learning and decision making with “representational” analysis methods that allow inferences to be made about the information represented in brain areas, applied to fMRI and scalp EEG data. fMRI will reveal how neural learning signals update neural representations of likely causes during learning, while EEG will reveal the timing of the hypothesized reinstatement. These experiments will set the stage to apply the insights gained to investigate how the brain attributes outcomes to more abstract “latent” causes in hierarchically structured environments prevalent in the real world. The proposed project will thus move this general program of research strategy toward learning tasks that better reflect the complexity and structure in many real-world learning/choice situations important for both typical and atypical individuals.