Decision dynamics during a continuous-time foraging task: a reinforcement learning approach
Decision dynamics during a continuous-time foraging task: a reinforcement learning approach
批准号:
10373999
负责人:
Benjamin Nicolaas Ballintyn
金额:
$0.52万
依托单位:
依托单位国家:
美国
项目类别:
财政年份:
2020
资助国家:
美国
项目状态:
已结题
起止时间:
2020-04-01 至 2022-04-15
关键词:
AlgorithmsAnimal BehaviorAnimalsBehaviorBehavioralBehavioral ModelBrainBreathingComplementComputer ModelsConsumptionDataDecision MakingDecision ModelingDevelopmentEconomicsElectrophysiology (science)Energy IntakeEnvironmentEvolutionFamilyFoodFree WillFutureGoalsHourHumanIndividualKnowledgeLearningLearning DisordersLifeLocationMeasurementMeasuresMemoryMethodsModelingNatural SelectionsNaturePalatePathologyPerformancePoliciesProbabilityProcessPsychological reinforcementRattusResearchResourcesRewardsSamplingSelf-control as a personality traitSourceStimulusStructureSystemTestingThirstTimeTravelUpdateWait TimeWeightaddictionbasebehavior predictioncognitive processdecision making algorithmdiscountingexperimental studyimprovedindexinginsightlearning algorithmlearning strategymotor disorderneural circuitneurophysiologypressurerelating to nervous systemreward circuitrysuccesstheories
中文摘要
项目摘要
很可能,进化已经强烈地塑造了奖励系统的神经电路,以优化
在寻找资源所涉及的许多任务中表现出色,这是每种动物生活中至关重要的一部分。这
这一命题启发了“最优”觅食理论的发展,例如边际
价值定理(MVT),它解析地推导出
最大化长期奖励率,通常被认为是能量摄入量。虽然这些分析
理论在描述动物行为方面取得了一些成功,这些理论本身依赖于严格的
对环境的假设在许多自然情况下都不成立,也不够灵活
推广到更复杂的环境或其他任务。因此,这个项目的最终目标是
了解通用决策(强化学习)算法家族中的哪一种最有可能
被大脑用来解决基于价值的任务,并使用这一知识来预测
在某些情况下,这些算法将导致最优或次优行为。
通过这个项目,我将提高我们对这些更自然的动物决策过程的理解
通过执行在时间上连续的觅食实验来保护环境,这违反了许多
先前的觅食分析理论所依赖的假设。被口渴刺激的老鼠将被允许
在开阔的田野上自由地从两个或三个(可口的或令人厌恶的)美味的选择(“补丁”)中进行品尝,
关键的是,他们将被允许直接接触选项,这是过去的实验所拥有的
缺乏。对每个品尝选项的舔(消费)行为的测量将使我能够
在几个1小时的会话中测量大鼠的决策动态。特别是,我将衡量
每个选项的采样时间与备选方案的价值相关,以深入了解老鼠如何
结合可用选项的值来做出决策。
作为对此行为任务的补充,我将模拟一组强化学习代理,这些代理在
用于学习操作值、选择操作和计划操作的规则。通过数量上
将这些人工制剂的决策行为与从老鼠身上获得的决策行为进行比较,我会确定哪一种
模拟的代理最好地再现了老鼠的行为,使人们能够深入了解
并为今后的电生理记录提供了方向。重要的是,这
将动物行为与人工制剂产生的行为进行比较,将使我能够评估
“最优”的老鼠行为是,在它不是最优的情况下,为
为什么会这样呢?
英文摘要
Project Summary
It is likely that evolution has strongly shaped the neural circuitry of the reward systems to optimize
performance in the many tasks involved in foraging for resources, a critical part of every animal's life. This
proposition was the inspiration for the development of “optimal” foraging theories, such as the marginal
value theorem (MVT), which derive analytically the foraging behavior (sequences of choices) that
maximizes the long-term rate of reward, usually considered to be energy intake. While these analytic
theories have had some success in describing animal behavior, the theories themselves rely on strict
assumptions about the environment that do not hold in many natural situations and are not flexible enough
to generalize to more complicated environments or other tasks. Therefore, the end goal of this project is to
understand which of a family of general-purpose decision (reinforcement-learning) algorithms is most likely
to be employed by the brain to solve value-based tasks and to use this knowledge to predict under what
circumstances these algorithms will lead to optimal or suboptimal behavior.
With this project, I will improve our understanding of animal decision processes in these more natural
environments by performing a foraging experiment that is continuous in time and violates many of the
assumptions that prior analytical theories of foraging rely on. Rats motivated by thirst will be allowed to
sample freely from two or three (palatable or aversive) tastant options (“patches”) in an open field and,
critically, will be allowed to direct their encounters with the options, something which past experiments have
lacked. Measurements of licking (consumption) behavior at each of the tastant options will allow me to
measure the decision dynamics of the rat over several 1-hour sessions. In particular, I will measure how the
sampling times at each option correlate with the values of the alternatives to gain insight into how rats
combine the values of available options to make decisions.
As a complement to this behavioral task, I will simulate a set of reinforcement learning agents that vary in
the rules used for learning action values, choosing actions, and planning actions. By quantitatively
comparing the decision behavior of these artificial agents to that obtained from rats I will determine which of
the simulated agents best reproduces the rat behavior, giving insight into the decision algorithms used by
rats and providing a direction for future electrophysiological recordings during this task. Importantly, this
comparison of animal behavior with that produced by artificial agents will allow me to assess how close to
“optimal” rat behavior is and, in the cases where it is suboptimal, to provide quantitative explanations for
why it is so.
期刊论文(0)
专著(0)
科研奖励(0)
会议论文
海外基金