Towards an Autonomous Brain Machine Interface: Integrating Sensorimotor Reward Modulation and Reinforcement Learning

 描述(由申请人提供)：目前的脑机接口(BMI)技术存在一个根本的差距，缺乏BMI在现实世界中工作的能力，以及与用户适应的能力。我们建议生产一种自主适应的BMI。我们工作的一个关键科学方面是进一步阐明我们关于报酬调节的初步发现 初级运动皮质(M1)。这一发现表明，我们可以使用M1中的一个植入物来生成我们的完全自主的BMI，并使用强化学习来更新BMI。人类目前正在被植入M1，使用与我们使用的相同的电极阵列，因此我们可能能够在短期内测试我们的系统。我们的长期生物医学工程目标是开发一种完全集成的BMI，使其用户能够识别系统本身，并做出看起来自然、准确的动作。这最终将需要一个与用户适应并提供感官反馈的系统。我们的长期神经科学目标是确定初级感觉运动皮质的基本特性，例如奖赏对这些区域内突触可塑性的影响以及感觉运动适应和学习。这个建议的目标有三个，我们将证明监督强化学习(RL)导致稳健的BMI，M1中存在奖励调制，并量化这种调制如何影响更传统的M1表征，如运动运动学。我们的中心假设是，奖励调节M1神经活动，这种调制可以被利用来实现自主的BMI，它将根据用户对BMI表现的解释来学习与用户合作。这一假设是由我们的初步工作推动的，该工作表明，强化学习方法可以产生良好的BMI控制，并且奖励调制对M1活动有影响。我们已经破译了M1在到达运动和被动观察过程中的奖励期望，然而，这是在奖励和非奖励之间，并没有像我们现在提出的那样对动机进行明确的控制。我们的具体目标是1)检验奖赏调制是否影响初级运动皮质对运动和动作观察的表征；2)检验在BMI控制下初级运动皮质是否存在奖赏调制的影响；3)检验带有神经衍生评估信号的监督强化学习在BMI控制中的应用。在我们看来，我们的贡献将是巨大的，因为它们将导致增强的BMI，学习与用户合作以提高整体性能，同时显著增加我们对M1的了解。这应该会增加BMI用户的独立性，并最终降低医疗保健成本。更不用说生活质量的提高了。在我们看来，这项拟议的研究具有创新性，因为它推动了将BMI从实验室转移到现实世界的能力，能够自动应对不断变化的环境。我们提议的工作将有助于将我们推向新的地平线，开启人类和利他主义计算代理之间的合作时代。

 DESCRIPTION (provided by applicant): There is a fundamental gap in current Brain Machine Interface (BMI) technology, the lack of BMIs' abilities to work in real world situations, and to adapt with the user. We suggest the production of an autonomously adapting BMI. A key scientific aspect of our work is to further elucidate our preliminary findings on reward modulation of the primary motor cortex (M1). This finding indicates that we could generate our fully autonomous BMI using one implant in M1 with the use of reinforcement learning to update the BMI. Humans are currently being implanted in M1 with the same electrode arrays we are using, and thus we may be able to test our system in the short term. Our long-term biomedical engineering goal is to develop a fully integrated BMI that allows its user to recognize the system as self and make natural looking accurate movements. This will ultimately require a system that adapts with the user and provides sensory feedback. Our long-term neuroscience goals are to determine the basic properties of the primary sensorimotor cortices, such as the influence of reward on synaptic plasticity within these regions and sensorimotor adaptation and learning. The goals of this proposal are threefold, we will prove that supervised reinforcement learning (RL) leads to robust BMIs, that there is reward modulation in M1 and quantify how this modulation influences more traditional M1 representations, such as movement kinematics. Our Central Hypothesis is that reward modulates M1 neural activity and this modulation can be tapped into for the purpose of an autonomous BMI, which will learn to work with the user based on the users interpretation of the BMIs performance. This hypothesis is driven by our preliminary work showing that reinforcement-learning methods can produce good BMI control and that reward modulation has an influence on M1 activity. We have decoded reward expectation from M1 during reaching movements and passive observation, however, this was between reward and non-reward, and did not control for motivation explicitly as we now propose. Our specific aims are 1) examine if reward modulation has an influence on the primary motor cortex's representation of movement and action observation; 2) examine if there is an influence of reward modulation on the primary motor cortex under BMI control; and 3) examine the use of supervised reinforcement learning with a neurally derived evaluative signal for BMI control. Our contributions will be significant, in our opinion, because they will lead to enhanced BMIs that learn to work with the user for improved overall performance while adding significantly to our knowledge on M1. This should lead to increased independence for the BMI's user, and ultimately a decrease in health care costs. Not to mention an increased quality of life. The proposed research is innovative, in our opinion, because it pushes to move BMIs from the lab to real world capabilities, able to deal with changing environments autonomously. Our proposed work will help push us to new horizons ushering in the age of teaming between humans and altruistic computational agents.