Towards an Autonomous Brain Machine Interface: Integrating Sensorimotor Reward Modulation and Reinforcement Learning

迈向自主脑机接口：整合感觉运动奖励调节和强化学习

• 批准号: 9332480
• 负责人: JOSEPH T FRANCIS
• 金额: $ 33.29万
• 依托单位: UNIVERSITY OF HOUSTON
• 依托单位国家: 美国
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-05-01 至 2019-04-30
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/9332480
• 关键词: Age; Altruism; Amputation; Architecture; Biomedical Engineering; Brain; Brain region; Computer Interface; Computers; Data; Derivation procedure; Dopamine Receptor; Electrodes; Environment; Goals; Hand; Health Care Costs; Human; Hybrids; Implant; Individual; Lead; Learning; Limb Prosthesis; Motion; Motivation; Motor; Motor Cortex; Movement; Muscarinic M1 Receptor; Neurologic; Neurosciences; Nucleus Accumbens; Performance; Production; Property; Psychological reinforcement; Public Health; Quality of life; Research; Rewards; Scientist; Signal Transduction; Structure; Supervision; Synaptic plasticity; System; Technology; Testing; Translating; Update; Vascular Diseases; Work; arm; base; brain machine interface; expectation; experimental study; feeding; grasp; improved; innovation; kinematics; learning strategy; nervous system disorder; neural correlate; public health relevance; relating to nervous system; robot control; sensory feedback; ward

 DESCRIPTION (provided by applicant): There is a fundamental gap in current Brain Machine Interface (BMI) technology, the lack of BMIs' abilities to work in real world situations, and to adapt with the user. We suggest the production of an autonomously adapting BMI. A key scientific aspect of our work is to further elucidate our preliminary findings on reward modulation of the primary motor cortex (M1). This finding indicates that we could generate our fully autonomous BMI using one implant in M1 with the use of reinforcement learning to update the BMI. Humans are currently being implanted in M1 with the same electrode arrays we are using, and thus we may be able to test our system in the short term. Our long-term biomedical engineering goal is to develop a fully integrated BMI that allows its user to recognize the system as self and make natural looking accurate movements. This will ultimately require a system that adapts with the user and provides sensory feedback. Our long-term neuroscience goals are to determine the basic properties of the primary sensorimotor cortices, such as the influence of reward on synaptic plasticity within these regions and sensorimotor adaptation and learning. The goals of this proposal are threefold, we will prove that supervised reinforcement learning (RL) leads to robust BMIs, that there is reward modulation in M1 and quantify how this modulation influences more traditional M1 representations, such as movement kinematics. Our Central Hypothesis is that reward modulates M1 neural activity and this modulation can be tapped into for the purpose of an autonomous BMI, which will learn to work with the user based on the users interpretation of the BMIs performance. This hypothesis is driven by our preliminary work showing that reinforcement-learning methods can produce good BMI control and that reward modulation has an influence on M1 activity. We have decoded reward expectation from M1 during reaching movements and passive observation, however, this was between reward and non-reward, and did not control for motivation explicitly as we now propose. Our specific aims are 1) examine if reward modulation has an influence on the primary motor cortex's representation of movement and action observation; 2) examine if there is an influence of reward modulation on the primary motor cortex under BMI control; and 3) examine the use of supervised reinforcement learning with a neurally derived evaluative signal for BMI control. Our contributions will be significant, in our opinion, because they will lead to enhanced BMIs that learn to work with the user for improved overall performance while adding significantly to our knowledge on M1. This should lead to increased independence for the BMI's user, and ultimately a decrease in health care costs. Not to mention an increased quality of life. The proposed research is innovative, in our opinion, because it pushes to move BMIs from the lab to real world capabilities, able to deal with changing environments autonomously. Our proposed work will help push us to new horizons ushering in the age of teaming between humans and altruistic computational agents.

 描述（由应用程序提供）：当前的脑机界面（BMI）技术存在基本差距，缺乏BMI在现实世界中工作的能力以及与用户适应的能力。我们建议生产自动调整BMI。我们工作的关键科学方面是进一步阐明我们的初步发现 一级运动皮层（M1）。这一发现表明，我们可以使用M1中的一种植入物生成完全自主的BMI，并使用增强学习来更新BMI。目前，人类正在使用与我们使用的相同电极阵列植入M1，因此我们可以在短期内测试我们的系统。我们的长期生物医学工程目标是开发一个完全集成的BMI，该BMI使其用户可以将系统识别为自我，并使自然看起来精确运动。最终将需要一个与用户适应的系统并提供感觉反馈。我们的长期神经科学目标是确定主要感觉运动皮层的基本特性，例如奖励对这些区域内合成可塑性的影响以及感觉运动适应和学习。该提案的目标是三倍，我们将证明受监督的强化学习（RL）导致BMIS稳健，M1中有奖励调制，并量化该调制如何影响更传统的M1表示，例如运动运动学运动。我们的核心假设是奖励调制M1神经元活动，并且可以将此调节列入为自主BMI的目的，该单位将根据用户对BMIS性能的解释来学会与用户合作。这一假设是由我们的初步工作驱动的，表明加强学习方法可以产生良好的BMI控制，而奖励调制对M1活性有影响。我们已经在达到运动和被动观察过程中对M1的奖励期望解码，但是，这是奖励和非回报之间的，并且没有像我们现在提出的那样明确控制动机。我们的具体目的是1）考试，如果奖励调制对主要运动皮层的运动和动作观察影响影响； 2）考试是否存在奖励调制对BMI控制下主要运动皮层的影响； 3）检查具有自然得出的评估信号的BMI控制信号的监督增强学习的使用。在我们看来，我们的贡献将非常重要，因为它们将导致增强的BMI，这些BMI学会与用户合作以提高整体性能，同时大大增加了我们对M1的知识。这将导致BMI用户的独立性增加，并最终降低医疗保健成本。更不用说增加的生活质量了。在我们看来，拟议的研究具有创新性，因为它促使BMI从实验室转移到现实世界的能力，能够自动处理不断变化的环境。我们提出的工作将有助于我们进入新的视野，并在人类与无私计算代理之间的团队时代迎来。