Embodied Decision Making: The Influence of Action Errors on Reinforcement Learning

 描述(由申请人提供)：本提案探讨了动作选择和动作执行中涉及的流程之间的交互作用。这种相互作用对于理解人们如何学习做出最佳决定和发展复杂技能至关重要，也对于解释运动系统障碍如何影响认知至关重要，这是小脑和基底节退行性疾病研究中的一个核心问题。考虑到成功的决策需要(至少)两种基本能力，这种互动是可以理解的。首先，代理必须能够评估环境中不同选项的价值，使用这些信息来选择将获得最大回报的选项。其次，代理必须能够执行响应以指示所选选项。传统的决策模型侧重于前者，而忽视了后者。然而，在许多现实世界中，执行中的错误是成功结果的主要障碍。网球运动员可能会正确地选择反手挥杆而不是正手回击发球，但却无法正确地执行动作。或者，在一个更普通的例子中，一个人可能会选择用酒杯而不是水杯喝一口，但却无法获得预期的奖励，因为她笨手笨脚地伸手碰翻了杯子。在这个例子中，问题是一个人在笨拙的伸手后是否不那么重视葡萄酒(由于未能获得预期的增援)，或者错误是否归因于执行系统，结果不会影响未来的选择行为(假设我们同样有能力伸手拿水或葡萄酒)。这项拟议的工作将研究动作执行和动作选择相互作用的心理过程和神经系统。为此，将解决两个具体目标。1)我们的试点工作表明，根据失败的结果是归因于对象的属性还是执行系统的限制，在选择行为上存在显著差异。将开发一系列有可能解释这种差异的计算模型。以共生的方式，一系列实验的行为数据将被用来评估这些模型，这些模型将被用来生成和测试特定的预测。2)确定动作错误和选择过程相互作用的神经区域。这里特别值得注意的是，基于小脑的动作执行错误表征可能具有双重作用，既改善了未来的动作执行，又提供了一个门控信号来限制作为动作选择动力学基础的学习过程。为了达到这一目的，实验计划需要综合使用功能成像和神经心理学研究。在这个项目完成后，这些研究将有助于提供一幅关于动作选择和动作执行过程如何在人脑中相互作用以优化行为的综合图景。

 DESCRIPTION (provided by applicant): This proposal explores the interaction of processes involved in action selection and action execution. This interaction is essential for understanding how people learn to make optimal decisions and develop complex skills, as well as for explicating how disorders of the motor system may impact cognition, a question that has been of central interest in studies of degenerative diseases of the cerebellum and basal ganglia. The interaction can be appreciated by considering that successful decision making requires (at least) two fundamental abilities. First, an agent must be able to evaluate the value of different options in the environment, using that information to choose the option that will maximize reward. Second, the agent must be able to execute a response to indicate the selected option. Traditionally, models of decision making have focused on the former and ignored the latter. However, in many real-world situations, errors in execution are the primary impediment to successful outcomes. A tennis player may correctly opt to use a backhand swing instead of a forehand to return a serve, but fail to execute the action properly. Or in a more mundane example, a person might choose to take a sip from the wine glass rather than the water glass, but fail to reap the expected reward because she knocks over the glass by reaching in a clumsy manner. In this example, the issue is whether a person values wine less (due to the failure to obtain the expected reinforcement) after the clumsy reach, or whether the error is attributed to the execution system, with the outcome precluded from influencing future choice behavior (assuming we are equally competent in reaching for water or wine). The proposed work will examine the psychological processes and neural systems through which action execution and action selection interact. To this end, two specific aims will be addressed. 1) Our pilot work demonstrates a striking difference in choice behavior depending on whether failed outcomes are attributed to a property of the object or a limitation of the execution system. A series of computational models will be developed that have the potential to account for this difference. In a symbiotic manner, behavioral data from a series of experiments will be used to evaluate the models, and the models will be used to generate and test specific predictions. 2) Identify the neural regions involved in the interaction of action errors and selection processes. Of special note here is the idea that cerebellar-based representations of action execution errors might serve a dual-purpose, improving future action execution and providing a gating signal to constrain learning processes that underlie action selection dynamics. The experimental plan for this aim entails the integrated use of functional imaging and neuropsychological studies. At the completion of this project, the studies will help provide an integrated picture of how action selection and action execution processes interact in the human brain to optimize behavior.