CPS: Synergy: A Novel Biomechatronic Interface Based on Wearable Dynamic Imaging Sensors

CPS：Synergy：基于可穿戴动态成像传感器的新型生物机电接口

• 批准号: 1329829
• 负责人: Siddhartha Sikdar
• 金额: $ 99.51万
• 依托单位: George Mason University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2014
• 资助国家: 美国
• 起止时间: 2014-02-01 至 2019-01-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1329829&HistoricalAwards=false
• 关键词: CPS; Synergy; Novel; Biomechatronic; Interface

The problem of controlling biomechatronic systems, such as multiarticulating prosthetic hands, involves unique challenges in the science and engineering of Cyber Physical Systems (CPS), requiring integration between computational systems for recognizing human functional activity and intent and controlling prosthetic devices to interact with the physical world. Research on this problem has been limited by the difficulties in noninvasively acquiring robust biosignals that allow intuitive and reliable control of multiple degrees of freedom (DoF). The objective of this research is to investigate a new sensing paradigm based on ultrasonic imaging of dynamic muscle activity. The synergistic research plan will integrate novel imaging technologies, new computational methods for activity recognition and learning, and high-performance embedded computing to enable robust and intuitive control of dexterous prosthetic hands with multiple DoF. The interdisciplinary research team involves collaboration between biomedical engineers, electrical engineers and computer scientists. The specific aims are to: (1) research and develop spatio-temporal image analysis and pattern recognition algorithms to learn and predict different dexterous tasks based on sonographic patterns of muscle activity (2) develop a wearable image-based biosignal sensing system by integrating multiple ultrasound imaging sensors with a low-power heterogeneous multicore embedded processor and (3) perform experiments to evaluate the real-time control of a prosthetic hand.The proposed research methods are broadly applicable to assistive technologies where physical systems, computational frameworks and low-power embedded computing serve to augment human activities or to replace lost functionality. The research will advance CPS science and engineering through integration of portable sensors for image-based sensing of complex adaptive physical phenomena such as dynamic neuromuscular activity, and real-time sophisticated image understanding algorithms to interpret such phenomena running on low-power high performance embedded systems. The technological advances would enable practical wearable image-based biosensing, with applications in healthcare, and the computational methods would be broadly applicable to problems involving activity recognition from spatiotemporal image data, such as surveillance.This research will have societal impacts as well as train students in interdisciplinary methods relevant to CPS. About 1.6 million Americans live with amputations that significantly affect activities of daily living. The proposed project has the long-term potential to significantly improve functionality of upper extremity prostheses, improve quality of life of amputees, and increase the acceptance of prosthetic limbs. This research could also facilitate intelligent assistive devices for more targeted neurorehabilitation of stroke victims. This project will provide immersive interdisciplinary CPS-relevant training for graduate and undergraduate students to integrate computational methods with imaging, processor architectures, human functional activity and artificial devices for solving challenging public health problems. A strong emphasis will be placed on involving undergraduate students in research as part of structured programs at our institution. The research team will involve students with disabilities in research activities by leveraging an ongoing NSF-funded project. Bioengineering training activities will be part of a newly developed undergraduate curriculum and a graduate curriculum under development.The synergistic research plan has been designed to advance CPS science and engineering through the development of new computational methods for dynamic activity recognition and learning from image sequences, development of novel wearable imaging technologies including high-performance embedded computing, and real-time control of a physical system. The specific aims are to: (1) Research and develop spatio-temporal image analysis and pattern recognition algorithms to learn and predict different dexterous tasks based on sonographic patterns of muscle activity. The first aim has three subtasks designed to collect, analyze and understand image sequences associated with functional tasks. (2) Develop a wearable image-based biosignal sensing system by integrating multiple ultrasound imaging sensors with a low-power heterogeneous multicore embedded processor. The second aim has two subtasks designed to integrate wearable imaging sensors with a real-time computational platform. (3) Perform experiments to evaluate the real-time control of a prosthetic hand. The third aim will integrate the wearable image acquisition system developed in Aim 2, and the image understanding algorithms developed in Aim 1, for real-time evaluation of the control of a prosthetic hand interacting with a virtual reality environment.Successful completion of these aims will result in a real-time system that acquires image data from complex neuromuscular activity, decodes activity intent from spatiotemporal image data using computational algorithms, and controls a prosthetic limb in a virtual reality environment in real time. Once developed and validated, this system can be the starting point for developing a new class of sophisticated control algorithms for intuitive control of advanced prosthetic limbs, new assistive technologies for neurorehabilitation, and wearable real-time imaging systems for smart health applications.

控制生物机电系统（例如多关节假手）的问题涉及信息物理系统（CPS）科学和工程中的独特挑战，需要计算系统之间的集成以识别人类功能活动和意图并控制假肢设备与物理世界交互。由于难以以非侵入方式获取鲁棒的生物信号，从而实现对多个自由度 (DoF) 的直观、可靠的控制，因此对该问题的研究受到了限制。本研究的目的是研究一种基于动态肌肉活动超声成像的新传感范例。该协同研究计划将整合新颖的成像技术、用于活动识别和学习的新计算方法以及高性能嵌入式计算，以实现对具有多个自由度的灵巧假手的鲁棒和直观控制。这个跨学科研究团队涉及生物医学工程师、电气工程师和计算机科学家之间的合作。具体目标是：（1）研究和开发时空图像分析和模式识别算法，以根据肌肉活动的超声模式学习和预测不同的灵巧任务（2）通过将多个超声成像传感器与低功耗异构多核嵌入式处理器集成来开发可穿戴的基于图像的生物信号传感系统，以及（3）进行实验以评估假肢的实时控制 所提出的研究方法广泛适用于辅助技术，其中物理系统、计算框架和低功耗嵌入式计算用于增强人类活动或替代失去的功能。该研究将通过集成便携式传感器来推进 CPS 科学和工程，这些传感器用于对复杂的自适应物理现象（例如动态神经肌肉活动）进行基于图像的传感，以及实时复杂的图像理解算法来解释在低功耗高性能嵌入式系统上运行的此类现象。这些技术进步将使基于图像的可穿戴生物传感技术在医疗保健领域得到应用，并且计算方法将广泛适用于涉及时空图像数据活动识别的问题，例如监控。这项研究将产生社会影响，并培训学生与 CPS 相关的跨学科方法。大约 160 万美国人遭受截肢，严重影响日常生活活动。该项目具有显着改善上肢假肢功能、提高截肢者生活质量并提高假肢接受度的长期潜力。这项研究还可以促进智能辅助设备对中风患者进行更有针对性的神经康复。该项目将为研究生和本科生提供身临其境的跨学科 CPS 相关培训，将计算方法与成像、处理器架构、人类功能活动和人工设备相结合，以解决具有挑战性的公共卫生问题。作为我们机构结构化课程的一部分，我们将非常重视让本科生参与研究。研究团队将利用正在进行的国家科学基金会资助的项目，让残疾学生参与研究活动。生物工程培训活动将成为新开发的本科课程和正在开发的研究生课程的一部分。协同研究计划旨在通过开发动态活动识别和图像序列学习的新计算方法、开发包括高性能嵌入式计算在内的新型可穿戴成像技术以及物理系统的实时控制来推进 CPS 科学和工程。具体目标是：（1）研究和开发时空图像分析和模式识别算法，以根据肌肉活动的超声模式来学习和预测不同的灵巧任务。第一个目标包含三个子任务，旨在收集、分析和理解与功能任务相关的图像序列。 (2)通过将多个超声成像传感器与低功耗异构多核嵌入式处理器集成，开发可穿戴的基于图像的生物信号传感系统。第二个目标有两个子任务，旨在将可穿戴成像传感器与实时计算平台集成。 (3) 进行实验评估假手的实时控制。第三个目标将集成目标 2 中开发的可穿戴图像采集系统和目标 1 中开发的图像理解算法，用于实时评估与虚拟现实环境交互的假手的控制。成功完成这些目标将产生一个实时系统，该系统从复杂的神经肌肉活动中获取图像数据，使用计算算法从时空图像数据中解码活动意图，并控制 虚拟现实环境中的实时假肢。一旦开发和验证，该系统可以成为开发新型复杂控制算法的起点，用于直观控制先进的假肢、用于神经康复的新辅助技术以及用于智能健康应用的可穿戴实时成像系统。

项目成果

An intuitive muscle-computer interface using ultrasound sensing and Markovian state transitions

使用超声波传感和马尔可夫状态转换的直观肌肉计算机界面

• DOI: 10.1109/isbi.2018.8363784
• 发表时间: 2018
• 期刊: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018
• 作者: Dhawan, Ananya S.;Kosecka, Jana;Rangwala, Huzefa;Sikdar, Siddhartha
• 通讯作者: Sikdar, Siddhartha