Creating Artificially intelligent neuroscience probes to determine how the brain makes decisions

强化学习（RL）模型通常用于研究生物学习和皮层下结构在这一过程中的作用，以及用于构建人工智能代理。然而，大脑中基于多巴胺的强化学习的经典概念并不能完全捕捉与人工智能强化学习模型相关的高阶过程。这种无模型的RL解释在计算上是便宜的，但有两个主要缺点：数据效率低，需要大量的经验来实现准确的估计，以及可扩展性，对结果值的变化不敏感。与哺乳动物认知的核心区别是它能够快速学习新的概念，从只有少数的例子，利用先验知识目标相关的“状态-值关联”，使灵活的归纳推理。越来越多的工作强调了新皮层对灵活决策的目标表征的重要性，因为需要开发新的模型来整合这些不同的目标驱动的行为和认知功能。能够同时成像，刺激和记录时空活动，从许多个别的神经元提供了一个有前途的框架，通过它来量化动物的行为，调查其与人口水平的神经元活动在自由移动的啮齿动物，以提供更好地了解哺乳动物大脑中的互补学习系统的关系。通过这些新兴的大脑高密度探测技术，神经科学家能够收集越来越庞大的数据集。这就提出了一个挑战，即如何有效地分析这些神经元记录并将其与观察到的行为相关联，以获得对生物学习和灵活决策的有意义的见解。 拟议的研究旨在通过利用头戴式微型镜的荧光共聚焦显微镜的多感官整合，神经元活动的电生理刺激/记录，以及在自由移动受试者的灵活学习研究中实时分析神经元数据的机器学习方法来解决这个问题。通过开发和实施复杂的行为分析，这项研究旨在发展对大脑的系统神经科学水平的理解，即它使用的算法，架构，功能和表示。这对应于理解任何复杂生物系统所需的最高两个分析层次：系统的目标（计算层次）和实现这一目标的过程（算法层次）。意义通过这些系统的组合，目的是促进刺激和记录方法的多模式控制，补充自由移动的啮齿动物进行行为实验的共聚焦钙成像。这一综合系统将允许通过一系列方式收集密集的数据集。利用实时数据分析，这将为进一步探索复杂认知背后的神经基础提供坚实的基础，从而为基于动态环境中生物学习的人工智能代理开发新的，灵活的强化学习模型。

Summary of Proposed Research Reinforcement learning (RL) models are often used for the study of biological learning and the role of subcortical structures in this process, as well as for building artificial intelligence agents. However, the classical concept of dopamine-based reinforcement learning in the brain does not fully capture the higher-order processes associated with reinforcement learning models of artificial intelligence. This model-free RL interpretation is computationally inexpensive but suffers from two primary drawbacks: data inefficiency, requiring large amounts of experience to achieve accurate estimates, and inflexibility, being insensitive to changes in the value of outcomes. A core distinction with mammalian cognition is its ability to rapidly learn new concepts from only a handful of examples, leveraging prior knowledge goal associated 'state-value associations' to enable flexible inductive inferences. A growing body of work has emphasized the importance of neocortical contributions towards goal representation with regards to flexible decision-making, as such new models need to be developed that incorporate these diverse goal-driven behavioural and cognitive functions. The ability to simultaneously image, stimulate and record spatiotemporal activity from many individual neurons provides a promising framework through which to quantify animal behaviour, investigating its relationship with population-level neuronal activity in freely moving rodents to provide better understanding of the complementary learning systems in the mammalian brain. Through these emerging technologies in high-density probing of the brain, neuroscientists are able to gather increasingly vast data sets. This presents the challenge of how to efficiently analyse and correlate these neuronal recordings with observed behaviour to gain meaningful insights into biological learning and flexible decision-making. The proposed research aims to address this issue through the multi-sensory integration of fluorescent confocal microscopy utilising head-mounted miniscopes, electrophysiological stimulation/recording of neuronal activity, and machine learning methodologies for the real-time analysis of neuronal data in the study of flexible learning in freely moving subjects. Through the development and implementation of complex behavioural assays, this research aims to develop a systems neuroscience-level understanding of the brain, namely the algorithms, architectures, functions and representations it utilizes. This corresponds to the top two levels of analysis believed to be required to understand any complex biological system: the goal of the system (computational level) & the processes that realize this goal (the algorithmic level). Significance Through the combination of these systems, the aim is to facilitate multi-modal control of stimulation and recording methodologies that complement confocal calcium imaging of freely-moving rodents performing behavioural experiments. This integrated system will allow for the collection of dense data sets through a range of modalities. Utilising real-time data analysis, this will provide a solid foundation through which to further explore the neural underpinnings behind complex cognition and thus develop new, flexible models of reinforcement learning for artificial intelligence agents based on biological learning in dynamic environments.