Interactions between motor exploration and reinforcement learning
Interactions between motor exploration and reinforcement learning
复制标题
DOI：
10.1152/jn.00390.2018
复制
发表时间：
2019-08-01
影响因子：
2.5
通讯作者：
Celnik, Pablo
中科院分区：
文献类型：
--
作者：
Uehara, Shintaro;Mawase, Firas;Celnik, Pablo
众所周知，运动探索是一种寻求更好运动结果的试错过程，在运动学习中发挥着关键作用。这在强化学习期间尤其重要，在强化学习中，导致成功结果的行动得到强化，同时避免不成功的行动。尽管早期的电机探索有利于找到正确的解决方案。在学习过程后期保持高水平的探索可能是有害的。然而，在强化学习过程中探索水平是否以及如何变化仍然知之甚少。在这里，我们评估了健康参与者学习基于强化的运动任务时运动探索的时间变化。我们将探索定义为每次试验的运动变化幅度，作为先前试验成功还是失败的函数。参与者被要求使用成功或失败的二元反馈找到最佳的指点方向。我们发现，当参与者学习任务时，探索的程度随着时间的推移逐渐增加。相反，对于无法正确调整指向方向的参与者来说，探索仍然很低。有趣的是，当参与者接受第二次训练时，探索程度仍然很高，这与更快的重新学习有关。这些结果表明，运动系统可以灵活地上调强化学习期间的探索程度，就好像获得了特定的策略来促进后续学习一样。此外，我们的研究结果表明，探索会影响强化学习，反之亦然，表明它们之间存在互动关系。基于强化的任务可以用作增加探索行为的引物，从而提高后续学习的效率。新的和值得注意的运动探索，即寻找正确动作的能力，对于学习运动技能至关重要。尽管如此，探索水平在训练过程中是否以及如何变化仍然知之甚少。我们表明，在基于强化的运动任务的训练过程中，探索不断增加并保持在较高水平。有趣的是，高水平的探索持续存在并促进了后续的学习。这些结果表明，运动系统会上调探索，就像学习一种促进后续学习的策略一样。
Motor exploration, a trial-and-error process in search for better motor outcomes, is known to serve a critical role in motor learning. This is particularly relevant during reinforcement learning, where actions leading to a successful outcome are reinforced while unsuccessful actions arc avoided. Although early on motor exploration is beneficial to finding the correct solution. maintaining high levels of exploration later in the learning process might be deleterious. Whether and how the level of exploration changes over the course of reinforcement learning, however, remains poorly understood. Here we evaluated temporal changes in motor exploration while healthy participants learned a reinforcement-based motor task. We defined exploration as the magnitude of trial-to-trial change in movements as a function of whether the preceding trial resulted in success or failure. Participants were required to find the optimal finger-pointing direction using binary feedback of success or failure. We found that the magnitude of exploration gradually increased over time when participants were learning the task. Conversely, exploration remained low in participants who were unable to correctly adjust their pointing direction. Interestingly, exploration remained elevated when participants underwent a second training session, which was associated with faster relearning. These results indicate that the motor system may flexibly upregulate the extent of exploration during reinforcement learning as if acquiring a specific strategy to facilitate subsequent learning. Also, our findings showed that exploration affects reinforcement learning and vice versa, indicating an interactive relationship between them. Reinforcement-based tasks could be used as primers to increase exploratory behavior leading to more efficient subsequent learning.NEW & NOTEWORTHY Motor exploration, the ability to search for the correct actions, is critical to learning motor skills. Despite this, whether and how the level of exploration changes over the course of training remains poorly understood. We showed that exploration increased and remained high throughout training of a reinforcement-based motor task. Interestingly, elevated exploration persisted and facilitated subsequent learning. These results suggest that the motor system upregulates exploration as if learning a strategy to facilitate subsequent learning.