Reinforcement learning in the human brain: Dimensions, features, and contexts

人类的决策不仅依赖于对世界的逻辑推理，还依赖于试错学习，将特征和行为与奖惩结果联系起来。这些强化学习机制具有众所周知的神经关联，这种学习理论为有限环境下的人类行为提供了极好的解释。然而，试错学习通常在实验室任务中进行研究，其中与价值相关的特征是已知的，并且是呈现给受试者的唯一刺激方面。因此，关于这种类型的学习如何在现实世界中完成的理论面临一个关键问题：任何给定的决定或结果都存在大量潜在的相关经验方面，那么应该学习哪些联系并指导未来的行为？这个项目探讨了人类在做出经验导向的决定时是如何应对多维世界的。更好地理解人类决策行为将促进更完整的学习理论，使我们能够更好地预测和增强现实世界的决策，并提高对这种决策如何因精神障碍而崩溃的理解。该项目将为本科生、研究生和博士后提供研究机会，并以在线演示和解释学习和决策模型的形式进行广泛的公众宣传。此外，拟议的活动还包括每年一次的核磁共振成像方法研讨会，这将为各个领域的学生和教师提供新方法的培训机会。该计划将使用行为的计算模型和基于模型的功能磁共振成像来评估人类如何在奖励引导的决策过程中了解世界的相关和不相关特征。第一项研究将探讨是否在选择行为和价值的神经表征方面追踪不相关的刺激维度。具体来说，该项目将询问腹侧纹状体和大脑其他部位的奖励预测错误信号是否仅由相关的特征值关联来解释，或者是否不相关的特征值关联也被神经跟踪并影响行为。第二项研究将检验在一个领域（视觉感知）中习得的统计偶然性是否在行为和大脑活动方面任意影响价值学习。这项研究的发现将阐明不同的联想学习机制如何相互作用来指导行为。最后，第三项研究将考察情境在奖励导向决策中的作用，考察情境特征如何很好地融入决策。这项工作的结果将指导人类强化学习理论的发展，以适应现实世界决策环境的复杂性。这个项目将阐明在学习或选择过程中，对特定联想指导行为的控制程度。

Human decision-making depends not only upon logical reasoning about the world, but also trial-and-error learning to associate features and actions with rewarding and punishing outcomes. These reinforcement-learning mechanisms have well-known neural correlates, and theories of such learning provide excellent accounts for human behavior in limited contexts. However, trial-and-error learning has typically been studied in laboratory tasks in which value-associated features are known and are the only stimulus aspects presented to subjects. Thus, theories of how this type of learning is accomplished in the real world face a key problem: there are a multitude of potentially relevant aspects of experience that co-occur with any given decision or outcome, so which associations should be learned and guide future behavior? This project explores how humans cope with a multidimensional world when making experience-guided decisions. A better understanding of human decision-making behavior will facilitate more complete theories of learning, enable us to better predict and enhance real-world decision-making, and improve understanding of how such decision-making might break down due to mental disorders. This project will provide research opportunities for undergraduate, graduate, and postdoctoral students, and include broad public outreach in the form of online demonstrations and explanations of models of learning and decision-making. Further, the proposed activity includes a yearly workshop on MRI methods, which will provide training opportunities in new methods and outreach to students and faculty across fields.The proposed project will use computational modeling of behavior and model-based fMRI to assess how humans learn about relevant and irrelevant features of the world during reward-guided decision-making. The first study will address whether irrelevant stimulus dimensions are tracked with respect to value, both in terms of choice behavior and in terms of neural representations of value. Specifically, the project will ask whether reward prediction error signals in ventral striatum and elsewhere in the brain are explained solely by relevant feature-value associations, or whether irrelevant feature-value associations are also tracked neurally and influence behavior. The second study will examine whether learned statistical contingencies in one domain, visual perception, arbitrarily influence value learning both in terms of behavior and brain activity. Findings from this study will illuminate how distinct associative learning mechanisms interact to guide behavior. Finally, a third study will examine the role of context in reward-guided decision-making, examining how well contextual features can be incorporated into decision-making. The results of this work will guide development of human reinforcement learning theories towards accommodating the complexity of real-world decision-making environments. This project will illuminate the degree to which control over which particular associations guide behavior is exerted during learning or at the time of choice.