AJILE Movement Prediction: Multimodal Deep Learning for Natural Human Neural Recordings and Video
AJILE Movement Prediction: Multimodal Deep Learning for Natural Human Neural Recordings and Video
复制标题
AJILE 运动预测：自然人类神经记录和视频的多模态深度学习
DOI：
--
复制
发表时间：
2017
期刊：
影响因子：
--
通讯作者：
Bingni W. Brunton
中科院分区：
文献类型：
--
作者：
X. Wang;Ali Farhadi;Rajesh P. N. Rao;Bingni W. Brunton
在大脑和机器之间开发有用的接口是神经工程学的一个巨大挑战。一个有效的界面不仅能够解释神经信号，而且能够预测人类在不久的将来执行某个动作的意图;在控制良好的实验室实验之外，预测变得更具挑战性。本文介绍了我们的方法来检测和预测自然的人类手臂运动在未来，大脑计算机接口的一个关键挑战，以前从未尝试过。我们介绍了新的长期ECoG（AJILE）数据集中的注释关节; AJILE包括4名人类受试者在总计670小时（超过7200万帧）内的7个上身关节的自动注释姿势，沿着相应的同时获取的颅内神经记录。AJILE的规模和范围大大超过了之前所有的运动和皮层电图（ECoG）数据集，使得采用深度学习方法进行运动预测成为可能。我们提出了一种多模态模型，将深度卷积神经网络（CNN）与长短期记忆（LSTM）块相结合，利用ECoG和视频模态。我们证明，我们的模型能够检测运动，并预测未来的运动开始前800毫秒。此外，我们的多模态运动预测模型对输入神经信号的模拟消融表现出弹性。我们认为，考虑到上下文的自然神经解码的多模式方法对于推进生物电子技术和人类神经科学至关重要。
Developing useful interfaces between brains and machines is a grand challenge of neuroengineering. An effective interface has the capacity to not only interpret neural signals, but predict the intentions of the human to perform an action in the near future; prediction is made even more challenging outside well-controlled laboratory experiments. This paper describes our approach to detect and to predict natural human arm movements in the future, a key challenge in brain computer interfacing that has never before been attempted. We introduce the novel Annotated Joints in Long-term ECoG (AJILE) dataset; AJILE includes automatically annotated poses of 7 upper body joints for four human subjects over 670 total hours (more than 72 million frames), along with the corresponding simultaneously acquired intracranial neural recordings. The size and scope of AJILE greatly exceeds all previous datasets with movements and electrocorticography (ECoG), making it possible to take a deep learning approach to movement prediction. We propose a multimodal model that combines deep convolutional neural networks (CNN) with long short-term memory (LSTM) blocks, leveraging both ECoG and video modalities. We demonstrate that our models are able to detect movements and predict future movements up to 800 msec before movement initiation. Further, our multimodal movement prediction models exhibit resilience to simulated ablation of input neural signals. We believe a multimodal approach to natural neural decoding that takes context into account is critical in advancing bioelectronic technologies and human neuroscience.
影响因子：
2.5
作者：
Jackson, Andrew;Mavoori, Jaideep;Fetz, Eberhard E.
通讯作者：
Fetz, Eberhard E.
使用高伽马皮质电图绘制自然上肢和下肢运动以及言语产生的体位图
影响因子：
5.7
作者：
Iljina;Johanna Ruescher;Dirk-Matthias Altenmüller;Ad Aertsen;Andreas Schulze-Bonhage ;Tonio Ball
通讯作者：
Tonio Ball