I-Corps: Dexterous Robotic Prosthetic Control Using Deep Learning Pattern Prediction from Ultrasound Signal

这个I-Corps项目的更广泛的影响/商业潜力在于开发一种新的系统，该系统将允许经桡动脉和部分手部截肢的人获得假肢手指运动的无与伦比的精确个性化，包括单个手指的连续和同时运动，而不需要长时间和复杂的训练过程。为了实现这样的功能，设计了一组新的深度学习算法来从超声图像中建模肌肉运动模式。该网络使用大量数据进行预训练，以尽量减少后期的单独训练。除了动力假肢之外，所提出的技术还可以在需要机器人和/或数字环境的易于使用和准确的手势控制的其他市场中产生广泛的影响。这些包括远程机器人、外骨骼操作、虚拟现实、游戏、手套箱以及与工作相关的个人防护设备和性能增强和放大设备。这个I-Corps项目将开发和利用一种新型的超声波传感器和新型的深度学习算法来识别连续的肌肉活动模式，从而预测准确和灵巧的手指运动。目前肌电驱动的假肢使用离散分类器，只能预测有限数量的离散手势从嘈杂的肌电图（EMG）信号。向卷积神经网络等深度学习架构提供丰富而详细的超声信号，有望实现对详细的连续和同时肌肉运动模式的建模和预测，这些模式可以映射到控制单个假肢手指的连续和同时运动。该项目的另一个优点是使用大量数据对这些深度神经网络进行预训练，这将允许个人用户进行短期微调训练，从而使该技术能够广泛而容易地采用。因此，拟议的项目可以让截肢者和上半身残疾者进行手指移动活动，如精细的物体操作，打字或演奏乐器。

The broader impact/commercial potential of this I-Corps project lies in the development a novel system that would allow people with transradial and partial hand amputations to gain unparalleled precise individuation of prosthetic digit motion including continuous and simultaneous movement for individual digits without requiring long and complicated training process. To allow for such functionality a novel set of deep learning algorithms are designed to model muscle movement patterns from ultrasound images. The network is pre-trained with a large amount of data in an effort to minimize later individual training. In addition to power prosthetics, the proposed technology can provide broad impacts in other markets where easy-to-use and accurate gestural control of robotics and/or digital environment are required. These include tele-robotics, exoskeleton operation, virtual reality, gaming, glove boxes as well as work related Personal Protective Equipment, and Performance Augmentation and Amplification Devices.This I-Corps project will develop and utilize a novel ultrasound sensor and novel deep learning algorithms to recognize continuous muscle activity patterns that can predict accurate and dexterous finger motion. Current myoelectric powered prostheses use discrete classifiers that can only predict a limited number of discrete gestures from noisy electromyography (EMG) signal. Feeding deep learning architectures, such as Convolutional Neural Networks, with rich and detailed ultrasound signal promises to allow for the modeling and prediction of detailed continuous and simultaneous muscle movements patterns, which can be mapped to control continuous and simultaneous movements of individual prosthetic fingers. An additional intellectual of this project merit is the pre-training of these deep neural network with a large amount of data, which would allow for short fine tuning training for individual users, allowing for wide and easy adoption of the technology. The proposed project could therefore allow amputees and people with upper body disabilities to perform finger-by-finger movement activities such as fine object manipulation, typing or playing a musical instrument.