CRCNS: Neural Basis of Planning

人类和其他动物可以使用多种学习算法和决策策略来选择自己的行为。例如，适应于稳定环境的习惯行为可以使用所谓的无模型强化学习算法来选择，在该算法中，每个动作的值根据意外奖励的量进行增量更新。这种强化学习的神经机制已经得到了深入的研究。相比之下，大脑如何利用动物对其环境的知识，通过基于模型的强化学习算法来计划连续的动作，仍未得到探索。在这项应用中，具有互补专业知识的PI将调查灵长类前额叶皮质的不同细分如何在灵长类动物的战略规划期间对不同学习算法的评估和仲裁做出贡献，使用一种称为“4 in-a-row”的顺序游戏。以前的研究表明，通过训练，人类通过逐渐从无模型的强化学习切换到基于模型的树搜索形式的强化学习来提高他们在这一游戏中的能力。在第一组实验中，我们将训练非人类灵长类动物与电脑对手玩4连排游戏。我们预测，战略规划的复杂性和对手违反动物预期的动作将反映在动物的动作速度和瞳孔直径上。接下来，我们将测试在动物和计算机对手之间的策略性交互中，前额叶皮质的内侧和外侧方面如何有助于评估和选择不同的学习算法。我们假设，外侧前额叶皮质参与计算来自多个来源的可选动作的积分值并指导动物的选择，而内侧前额叶皮质可能更多地参与监控和解决不同学习算法所支持的动作的差异。这些实验的结果将扩大我们对复杂战略规划的神经机制的了解，并统一研究自然行为的各种方法。通过利用机器学习和决策神经科学的最新进展，拟议的研究将阐明如何通过前额叶皮质的特定活动模式同时实施和协调多种学习算法。这些研究的结果将改变用于研究高阶计划及其在人类和动物中的神经基础的行为和分析范式。

Humans and other animals can choose their actions using multiple learning algorithms and decision­ making strategies. For example, habitual behaviors adapted to a stable environment can be selected using so-called model-free reinforcement learning algorithms, in which the value of each action is incrementally updated according to the amount of unexpected reward. The underlying neural mechanisms for this type of reinforcement learning have been intensively studied. By contrast, how the brain utilizes the animal's knowledge of its environment to plan sequential actions using a model-based reinforcement learning algorithm remains unexplored. In this application, PIs with complementary expertise will investigate how different subdivisions of the primate prefrontal cortex contribute to the evaluation and arbitration of different learning algorithms during strategic planning in primates, using a sequential game referred to as "4-in-a­ row". Previous studies have revealed that with training, humans improve their competence in this game by gradually switching away from a model-free reinforcement learning towards a model-based reinforcement learning in the form of a tree search. In the first set of experiments, we will train non-human primates to play the 4-in-a-row game against a computer opponent. We predict that the complexity of the strategic planning and the opponent's move violating the animal's expectation will be reflected in the speed of animal's action and pupil diameters. Next, we will test how the medial and lateral aspects of prefrontal cortex contribute to the evaluation and selection of different learning algorithms during strategic interaction between the animal and computer opponent. We hypothesize that the lateral prefrontal cortex is involved in computing the integrated values of alternative actions originating from multiple sources and guiding the animal's choice, whereas the medial prefrontal cortex might be more involved in monitoring and resolving the discrepancies of actions favored by different learning algorithms. The results from these experiments will expand our knowledge of the neural mechanisms for complex strategic planning and unify various approaches to study naturalistic behaviors. By taking advantage of recent advances in machine learning and decision neuroscience, proposed studies will elucidate how multiple learning algorithms are simultaneously implemented and coordinated via specific patterns of activity in the prefrontal cortex. The results from these studies will transform the behavioral and analytical paradigms used to study high-order planning and their neural underpinnings in humans and animals.