CPS: Synergy: A Novel Biomechatronic Interface Based on Wearable Dynamic Imaging Sensors

控制生物机电系统的问题，如多关节假手，在计算机物理系统(CPS)的科学和工程中涉及到独特的挑战，需要计算机系统之间的集成来识别人类的功能活动和意图，并控制假肢设备与物理世界交互。这一问题的研究一直受到非侵入性获取健壮生物信号的困难的限制，这些信号允许直观和可靠地控制多个自由度(DoF)。本研究的目的是探索一种新的基于动态肌肉活动超声成像的传感范式。协同研究计划将整合新的成像技术、用于活动识别和学习的新计算方法，以及高性能的嵌入式计算，以实现对具有多个DOF的灵巧假手的稳健和直观控制。跨学科研究团队涉及生物医学工程师、电气工程师和计算机科学家之间的合作。具体目标是：(1)研究和开发时空图像分析和模式识别算法，以学习和预测基于肌肉活动的超声模式的不同灵巧任务；(2)通过将多个超声成像传感器与低功耗异构多核嵌入式处理器相集成，开发基于可穿戴图像的生物信号传感系统；(3)进行实验以评估假手的实时控制。所提出的研究方法广泛适用于物理系统、计算框架和低功耗嵌入式计算用于增强人类活动或取代丧失的功能的辅助技术。这项研究将通过集成便携式传感器来对复杂的适应性物理现象(如动态神经肌肉活动)进行基于图像的传感，以及实时复杂的图像理解算法来解释运行在低功耗高性能嵌入式系统上的此类现象，从而推动CPS的科学和工程。这些技术进步将使基于可穿戴图像的生物传感在医疗保健中得到应用，计算方法将广泛适用于从时空图像数据中进行活动识别的问题，如监控。这项研究将产生社会影响，并培训学生使用与CPS相关的跨学科方法。约有160万美国人接受截肢，这严重影响了日常生活活动。拟议的项目具有显著改善上肢假体功能、改善截肢者的生活质量和提高假肢接受度的长期潜力。这项研究还可以促进智能辅助设备，为中风患者提供更有针对性的神经康复。该项目将为研究生和本科生提供身临其境的跨学科CPS相关培训，将计算方法与成像、处理器架构、人类功能活动和人造设备相结合，以解决具有挑战性的公共卫生问题。作为我们学院有组织的课程的一部分，我们将大力强调让本科生参与研究。研究团队将利用一个正在进行的国家科学基金会资助的项目，让残疾学生参与研究活动。生物工程培训活动将是新开发的本科课程和正在开发的研究生课程的一部分。已经设计了协同研究计划，通过开发用于从图像序列中动态活动识别和学习的新计算方法，开发包括高性能嵌入式计算在内的新型可穿戴成像技术，以及对物理系统的实时控制，来促进CPS科学和工程。具体目标是：(1)研究和开发基于肌肉活动超声模式的时空图像分析和模式识别算法，学习和预测不同的灵巧任务。第一个目标有三个子任务，旨在收集、分析和理解与功能任务相关的图像序列。(2)将多个超声成像传感器与低功耗的异质多核嵌入式处理器相结合，开发了一种可穿戴式的基于图像的生物信号传感系统。第二个目标有两个子任务，旨在将可穿戴式成像传感器与实时计算平台相结合。(3)对假手的实时控制进行了实验研究。第三个目标将集成Aim 2中开发的可穿戴图像采集系统和Aim 1中开发的图像理解算法，用于实时评估假手与虚拟现实环境交互的控制。成功完成这些目标将导致一个实时系统，该系统从复杂的神经肌肉活动中获取图像数据，使用计算算法从时空图像数据中解码活动意图，并在虚拟现实环境中实时控制假肢。一旦开发和验证，该系统可以成为开发用于直观控制高级假肢的新型复杂控制算法、用于神经康复的新辅助技术以及用于智能健康应用的可穿戴实时成像系统的起点。

The problem of controlling biomechatronic systems, such as multiarticulating prosthetic hands, involves unique challenges in the science and engineering of Cyber Physical Systems (CPS), requiring integration between computational systems for recognizing human functional activity and intent and controlling prosthetic devices to interact with the physical world. Research on this problem has been limited by the difficulties in noninvasively acquiring robust biosignals that allow intuitive and reliable control of multiple degrees of freedom (DoF). The objective of this research is to investigate a new sensing paradigm based on ultrasonic imaging of dynamic muscle activity. The synergistic research plan will integrate novel imaging technologies, new computational methods for activity recognition and learning, and high-performance embedded computing to enable robust and intuitive control of dexterous prosthetic hands with multiple DoF. The interdisciplinary research team involves collaboration between biomedical engineers, electrical engineers and computer scientists. The specific aims are to: (1) research and develop spatio-temporal image analysis and pattern recognition algorithms to learn and predict different dexterous tasks based on sonographic patterns of muscle activity (2) develop a wearable image-based biosignal sensing system by integrating multiple ultrasound imaging sensors with a low-power heterogeneous multicore embedded processor and (3) perform experiments to evaluate the real-time control of a prosthetic hand.The proposed research methods are broadly applicable to assistive technologies where physical systems, computational frameworks and low-power embedded computing serve to augment human activities or to replace lost functionality. The research will advance CPS science and engineering through integration of portable sensors for image-based sensing of complex adaptive physical phenomena such as dynamic neuromuscular activity, and real-time sophisticated image understanding algorithms to interpret such phenomena running on low-power high performance embedded systems. The technological advances would enable practical wearable image-based biosensing, with applications in healthcare, and the computational methods would be broadly applicable to problems involving activity recognition from spatiotemporal image data, such as surveillance.This research will have societal impacts as well as train students in interdisciplinary methods relevant to CPS. About 1.6 million Americans live with amputations that significantly affect activities of daily living. The proposed project has the long-term potential to significantly improve functionality of upper extremity prostheses, improve quality of life of amputees, and increase the acceptance of prosthetic limbs. This research could also facilitate intelligent assistive devices for more targeted neurorehabilitation of stroke victims. This project will provide immersive interdisciplinary CPS-relevant training for graduate and undergraduate students to integrate computational methods with imaging, processor architectures, human functional activity and artificial devices for solving challenging public health problems. A strong emphasis will be placed on involving undergraduate students in research as part of structured programs at our institution. The research team will involve students with disabilities in research activities by leveraging an ongoing NSF-funded project. Bioengineering training activities will be part of a newly developed undergraduate curriculum and a graduate curriculum under development.The synergistic research plan has been designed to advance CPS science and engineering through the development of new computational methods for dynamic activity recognition and learning from image sequences, development of novel wearable imaging technologies including high-performance embedded computing, and real-time control of a physical system. The specific aims are to: (1) Research and develop spatio-temporal image analysis and pattern recognition algorithms to learn and predict different dexterous tasks based on sonographic patterns of muscle activity. The first aim has three subtasks designed to collect, analyze and understand image sequences associated with functional tasks. (2) Develop a wearable image-based biosignal sensing system by integrating multiple ultrasound imaging sensors with a low-power heterogeneous multicore embedded processor. The second aim has two subtasks designed to integrate wearable imaging sensors with a real-time computational platform. (3) Perform experiments to evaluate the real-time control of a prosthetic hand. The third aim will integrate the wearable image acquisition system developed in Aim 2, and the image understanding algorithms developed in Aim 1, for real-time evaluation of the control of a prosthetic hand interacting with a virtual reality environment.Successful completion of these aims will result in a real-time system that acquires image data from complex neuromuscular activity, decodes activity intent from spatiotemporal image data using computational algorithms, and controls a prosthetic limb in a virtual reality environment in real time. Once developed and validated, this system can be the starting point for developing a new class of sophisticated control algorithms for intuitive control of advanced prosthetic limbs, new assistive technologies for neurorehabilitation, and wearable real-time imaging systems for smart health applications.