Towards an Autonomous Brain Machine Interface: Integrating Sensorimotor Reward Modulation and Reinforcement Learning

 描述（由申请人提供）：目前的脑机接口（BMI）技术存在根本性的差距，缺乏BMI在真实的世界环境中工作的能力，以及适应用户的能力。 我们建议生产自主适应BMI。 我们工作的一个关键科学方面是进一步阐明我们对奖励调节的初步发现 初级运动皮质（M1）。 这一发现表明，我们可以使用M1中的一个植入物生成完全自主的BMI，并使用强化学习来更新BMI。 目前，人类正在M1中植入与我们使用的相同的电极阵列，因此我们可能能够在短期内测试我们的系统。 我们的长期生物医学工程目标是开发一个完全集成的BMI，允许其用户将系统识别为自我，并做出自然的准确动作。 这最终需要一个能够适应用户并提供感官反馈的系统。 我们的长期神经科学目标是确定初级感觉运动皮层的基本特性，例如奖赏对这些区域内突触可塑性以及感觉运动适应和学习的影响。 该提案的目标有三个，我们将证明监督强化学习（RL）导致强大的BMI，M1中存在奖励调制，并量化这种调制如何影响更传统的M1表示，如运动学。 我们的中心假设是，奖励调节M1神经活动，这种调节可以用于自主BMI的目的，它将根据用户对BMI表现的解释学习与用户合作。 这一假设是由我们的初步工作驱动的，我们的初步工作表明，重复学习方法可以产生良好的BMI控制，奖励调节对M1活动有影响。 我们已经在到达运动和被动观察期间从M1解码了奖励期望，然而，这是在奖励和非奖励之间，并且没有像我们现在提出的那样明确地控制动机。 我们的具体目标是：1）检查奖励调制是否对初级运动皮层的运动和动作观察的表征产生影响; 2）检查奖励调制是否对BMI控制下的初级运动皮层产生影响; 3）检查监督强化学习与神经衍生的评估信号的使用。 我们认为，我们的贡献将是重大的，因为它们将导致增强的BMI，学习与用户一起工作，以提高整体性能，同时显著增加我们对M1的了解。 这应该会增加BMI用户的独立性，并最终降低医疗保健成本。 更不用说生活质量的提高了。在我们看来，拟议的研究是创新的，因为它推动了BMI从实验室到真实的世界的能力，能够自主处理不断变化的环境。 我们提出的工作将有助于推动我们进入新的视野，迎来人类和利他计算代理之间的合作时代。

 DESCRIPTION (provided by applicant): There is a fundamental gap in current Brain Machine Interface (BMI) technology, the lack of BMIs' abilities to work in real world situations, and to adapt with the user. We suggest the production of an autonomously adapting BMI. A key scientific aspect of our work is to further elucidate our preliminary findings on reward modulation of the primary motor cortex (M1). This finding indicates that we could generate our fully autonomous BMI using one implant in M1 with the use of reinforcement learning to update the BMI. Humans are currently being implanted in M1 with the same electrode arrays we are using, and thus we may be able to test our system in the short term. Our long-term biomedical engineering goal is to develop a fully integrated BMI that allows its user to recognize the system as self and make natural looking accurate movements. This will ultimately require a system that adapts with the user and provides sensory feedback. Our long-term neuroscience goals are to determine the basic properties of the primary sensorimotor cortices, such as the influence of reward on synaptic plasticity within these regions and sensorimotor adaptation and learning. The goals of this proposal are threefold, we will prove that supervised reinforcement learning (RL) leads to robust BMIs, that there is reward modulation in M1 and quantify how this modulation influences more traditional M1 representations, such as movement kinematics. Our Central Hypothesis is that reward modulates M1 neural activity and this modulation can be tapped into for the purpose of an autonomous BMI, which will learn to work with the user based on the users interpretation of the BMIs performance. This hypothesis is driven by our preliminary work showing that reinforcement-learning methods can produce good BMI control and that reward modulation has an influence on M1 activity. We have decoded reward expectation from M1 during reaching movements and passive observation, however, this was between reward and non-reward, and did not control for motivation explicitly as we now propose. Our specific aims are 1) examine if reward modulation has an influence on the primary motor cortex's representation of movement and action observation; 2) examine if there is an influence of reward modulation on the primary motor cortex under BMI control; and 3) examine the use of supervised reinforcement learning with a neurally derived evaluative signal for BMI control. Our contributions will be significant, in our opinion, because they will lead to enhanced BMIs that learn to work with the user for improved overall performance while adding significantly to our knowledge on M1. This should lead to increased independence for the BMI's user, and ultimately a decrease in health care costs. Not to mention an increased quality of life. The proposed research is innovative, in our opinion, because it pushes to move BMIs from the lab to real world capabilities, able to deal with changing environments autonomously. Our proposed work will help push us to new horizons ushering in the age of teaming between humans and altruistic computational agents.