Embodied Decision Making: The Influence of Action Errors on Reinforcement Learning

具身决策：行动错误对强化学习的影响

• 批准号: 9263024
• 负责人: RICHARD IVRY
• 金额: $ 34.03万
• 依托单位: UNIVERSITY OF CALIFORNIA BERKELEY
• 依托单位国家: 美国
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-04-01 至 2020-03-31
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/9263024
• 关键词: Address; Architecture; Attenuated; Basal Ganglia; Behavior; Behavioral; Biological; Brain; Cerebellar Diseases; Cerebellum; Cerebral cortex; Choice Behavior; Coffee; Cognition; Complement; Complex; Computer Simulation; Data; Decision Making; Degenerative Disorder; Employee Strikes; Environment; Event; Failure; Feedback; Functional Imaging; Functional Magnetic Resonance Imaging; Functional disorder; Future; Glass; Goals; Human; Inferior; Intuition; Laws; Learning; Lesion; Literature; Love; Methods; Modeling; Motor; Movement; Neurobiology; Neuropsychology; Outcome; Parietal Lobe; Parkinson Disease; Participant; Patients; Persons; Prefrontal Cortex; Process; Property; Psychological reinforcement; Research; Rewards; Role; Seminal; Sensory; Series; Signal Transduction; Source; Structure; System; Tennis; Testing; Update; Variant; Water; Wine; Work; arm; base; cognitive process; experimental study; frontal lobe; improved; interest; learning network; motor control; motor disorder; nervous system disorder; neuroimaging; neuromechanism; novel; psychologic; public health relevance; relating to nervous system; response; sensorimotor system; skills

 DESCRIPTION (provided by applicant): This proposal explores the interaction of processes involved in action selection and action execution. This interaction is essential for understanding how people learn to make optimal decisions and develop complex skills, as well as for explicating how disorders of the motor system may impact cognition, a question that has been of central interest in studies of degenerative diseases of the cerebellum and basal ganglia. The interaction can be appreciated by considering that successful decision making requires (at least) two fundamental abilities. First, an agent must be able to evaluate the value of different options in the environment, using that information to choose the option that will maximize reward. Second, the agent must be able to execute a response to indicate the selected option. Traditionally, models of decision making have focused on the former and ignored the latter. However, in many real-world situations, errors in execution are the primary impediment to successful outcomes. A tennis player may correctly opt to use a backhand swing instead of a forehand to return a serve, but fail to execute the action properly. Or in a more mundane example, a person might choose to take a sip from the wine glass rather than the water glass, but fail to reap the expected reward because she knocks over the glass by reaching in a clumsy manner. In this example, the issue is whether a person values wine less (due to the failure to obtain the expected reinforcement) after the clumsy reach, or whether the error is attributed to the execution system, with the outcome precluded from influencing future choice behavior (assuming we are equally competent in reaching for water or wine). The proposed work will examine the psychological processes and neural systems through which action execution and action selection interact. To this end, two specific aims will be addressed. 1) Our pilot work demonstrates a striking difference in choice behavior depending on whether failed outcomes are attributed to a property of the object or a limitation of the execution system. A series of computational models will be developed that have the potential to account for this difference. In a symbiotic manner, behavioral data from a series of experiments will be used to evaluate the models, and the models will be used to generate and test specific predictions. 2) Identify the neural regions involved in the interaction of action errors and selection processes. Of special note here is the idea that cerebellar-based representations of action execution errors might serve a dual-purpose, improving future action execution and providing a gating signal to constrain learning processes that underlie action selection dynamics. The experimental plan for this aim entails the integrated use of functional imaging and neuropsychological studies. At the completion of this project, the studies will help provide an integrated picture of how action selection and action execution processes interact in the human brain to optimize behavior.

 描述（由申请人提供）：本提案探讨了行动选择和行动执行过程中的相互作用。这种相互作用对于理解人们如何学习做出最佳决策和发展复杂技能以及解释运动系统疾病如何影响认知至关重要，这是小脑和基底神经节退行性疾病研究的核心问题。这种相互作用可以通过考虑成功的决策需要（至少）两种基本能力来理解。首先，智能体必须能够评估环境中不同选项的价值，利用这些信息来选择将最大化回报的选项。其次，代理必须能够执行响应以指示所选选项。传统上，决策模型关注前者而忽视后者。然而，在许多现实世界的情况下，执行中的错误是成功结果的主要障碍。一个网球运动员可以正确地选择使用反手挥杆而不是反手击球来回击发球，但不能正确地执行这个动作。或者在一个更平凡的例子中，一个人可能会选择从酒杯而不是水杯中啜饮一口，但却没有获得预期的奖励，因为她笨拙地伸手打翻了杯子。在这个例子中，问题是一个人在笨拙地伸手之后是否更不重视葡萄酒（由于未能获得预期的强化），或者错误是否归因于执行系统，结果被排除在影响未来选择行为之外（假设我们同样有能力伸手去拿水或葡萄酒）。拟议的工作将检查的心理过程和神经系统，通过行动执行和行动选择相互作用。为此，将探讨两个具体目标。1)我们的试点工作表明，选择行为的显着差异取决于失败的结果是否归因于对象的属性或执行系统的限制。将开发一系列计算模型，这些模型有可能解释这种差异。以共生的方式，来自一系列实验的行为数据将用于评估模型，模型将用于生成和测试特定的预测。2)识别参与动作错误和选择过程相互作用的神经区域。这里特别值得注意的是，基于小脑的动作执行错误表示可能具有双重目的，改善未来的动作执行，并提供门控信号来约束动作选择动态基础的学习过程。为此目的的实验计划需要综合使用功能成像和神经心理学研究。在这个项目完成后，这些研究将有助于提供一个综合的图片，如何行动选择和行动执行过程在人脑中相互作用，以优化行为。