CAREER: Instrumental divergence and goal-directed choice

工具性行为理论区分了目标导向的决策和习惯，前者是基于对结果的概率和当前效用的深思熟虑而做出的决定，后者是基于强化历史的刺激环境严格而自动地引发的。 尽管这一区别具有深远的意义，从经济政策的构建到行为病理的诊断和治疗，但对于什么因素塑造了目标导向的决策，以及什么条件促使从目标导向的行为选择转变为习惯性行为选择，我们仍然知之甚少。 一般来说，虽然计算成本很高，但目标导向策略提供了更高水平的灵活工具控制。 由于主观效用经常从一个时刻到下一个时刻发生变化，这种灵活性对于奖励最大化是必不可少的，因此可能具有内在价值，可能有助于激励和加强特定的决策，以及证明目标导向计算的一般处理成本。 然而，灵活的工具控制的一个关键要求是，可用的行动方案产生不同的结果状态。 在这个NSF职业奖的支持下，Mimi Liljeholm博士正在研究新的假设，即工具分歧？与替代行动相关的结果概率分布之间的差异？可以塑造选择偏好，诱导条件强化，并在目标导向和习惯性决策策略之间进行仲裁。 本研究的目的是解决目前的知识的性质和限制的目标导向行为的重要差距，使用创新的实验设计，计算建模和功能磁共振成像（fMRI）的组合。 该奖项的教育部分为本科生和研究生提供神经成像方法以及学习和决策的计算和神经基础方面的实践培训。 所有的研究都使用了一个简单的赌博任务，在这个任务中，不同的行为产生不同颜色的代币，每个代币价值特定的金额，具有不同的概率。 在评估灵活的工具控制偏好的研究中，相关的选择是在具有不同程度的工具分歧的成对行动之间。 预期的货币回报的变化独立的工具分歧的选择，分离的相对贡献，每个因素的行为选择的表现。 研究调查的能力，高工具分歧，以诱导条件强化措施的变化，在情感价的视觉刺激的基础上，他们与高与低工具分歧。 最后，在长期暴露于高与低工具分歧之后，使用标准的结果贬值程序来评估行为是目标导向的或习惯性的程度，其中与特定令牌颜色相关联的货币金额被改变：目标导向的，但不是习惯性的，决策受到特定感觉结果状态效用的这种变化的调制。 神经成像数据是通过在参与者执行任务时使用功能磁共振成像扫描来获取的，并且强化学习框架用于在行为和神经水平上对灵活工具控制的内在价值进行建模（通过将工具分歧视为替代奖励）。由于许多精神疾病的特点是一个异常的代理感，成瘾与目标导向的快速转变为习惯性的行动选择，更广泛的影响，这个项目包括潜在的发展的临床前诊断检测的早期发现的认知，情感和行为病理。 该项目提出的概念也可能有助于提高强化学习算法的性能，例如通过使用工具发散来指定新的优化标准，这可能有利于人工智能的医疗，工业和商业应用。

Theories of instrumental behavior distinguish between goal-directed decisions, motivated by a deliberate consideration of the probability and current utility of their consequences, and habits, which are rigidly and automatically elicited by the stimulus environment based on reinforcement history. In spite of the far-reaching implications of this distinction, ranging from the structuring of economic policies to the diagnosis and treatment of behavioral pathology, much is still unknown about what factors shape goal-directed decisions and what conditions prompt a transition from goal-directed to habitual action selection. Generally, while computationally expensive, a goal-directed strategy offers greater levels of flexible instrumental control. Since subjective utilities often change from one moment to the next, such flexibility is essential for reward maximization and thus may have intrinsic value, potentially serving to motivate and reinforce specific decisions, as well as to justify the general processing cost of goal-directed computations. A critical requirement for flexible instrumental control, however, is that available action alternatives yield distinct outcome states. With the support of this NSF Career award, Dr. Mimi Liljeholm is investigating the novel hypotheses that instrumental divergence? the difference between outcome probability distributions associated with alternative actions? can shape choice preferences, induce conditioned reinforcement, and arbitrate between goal-directed and habitual decision strategies. The objective of this research is to address important gaps in current knowledge about the nature and limits of goal-directed behavior, using a combination of innovative experimental designs, computational modeling and functional magnetic resonance imaging (fMRI). The educational component of the award provides hands-on training in neuroimaging methods, and in the computational and neural bases of learning and decision-making, at undergraduate and graduate levels. All studies use a simple gambling task in which alternative actions yield different colored tokens, each worth a particular amount of money, with various probabilities. In studies assessing a preference for flexible instrumental control, the relevant choice is between pairs of actions with different levels of instrumental divergence. Expected monetary pay-offs vary independently of instrumental divergence across options, dissociating the relative contribution of each factor to behavioral choice performance. Studies investigating the capacity of high instrumental divergence to induce conditioned reinforcement measure changes in the affective valence of visual stimuli based on their association with high versus low instrumental divergence. Finally, following extended exposure to high versus low instrumental divergence, the degree to which behavior is goal-directed or habitual is assessed using a standardly employed outcome devaluation procedure, in which the monetary amount associated with a particular token color is altered: Goal-directed, but not habitual, decisions are modulated by such changes in the utility of sensory-specific outcomes states. Neuroimaging data is acquired by scanning participants with fMRI as they perform the task, and a reinforcement learning framework is used to model the intrinsic value of flexible instrumental control (by treating instrumental divergence as a surrogate reward) at behavioral and neural levels. Since many psychiatric disorders are characterized by an abnormal sense of agency, and addiction associated with a rapid transition from goal-directed to habitual action-selection, broader impacts of this project include the potential development of pre-clinical diagnostic assays for early detection of cognitive, affective and behavioral pathology. The concepts advanced under this project may also help improve the performance of reinforcement learning algorithms, for example by using instrumental divergence to specify new optimization criteria, potentially benefiting medical, industrial and commercial applications of artificial intelligence.