Improving the understanding of neuromuscular gait control using deep reinforcement learning

使用深度强化学习提高对神经肌肉步态控制的理解

• 批准号: 456562029
• 负责人: Professor Jan Reinhard Peters, Ph.D.
• 金额: --
• 依托单位: Institut für Sportwissenschaft
• 依托单位国家: 德国
• 项目类别: Research Grants
• 资助国家: 德国
• 项目状态: 未结题
• 来源: https://gepris.dfg.de/gepris/projekt/456562029?language=en
• 关键词: Improving; understanding; neuromuscular; gait; control

Musculoskeletal disorders are one of the leading causes of human disability. The risk of such disabilities is increasing in many countries with an aging population like Germany. Assistive devices (e.g. exoskeletons, prosthesis, etc.) can facilitate human locomotor function and improve walking performance (e.g. balance, metabolic cost, etc.). It has been shown that human walking performance can be improved by the exoskeleton with subject specific control parameters using the human-in-the-loop (HIL) optimization approach. However, the HIL approach is limited in terms of gait diversity (e.g. different walking speeds) and optimization time (e.g. one hour of continuous walking on a treadmill). The large amount of time required to find the optimal parameters using a HIL-like method might not be feasible and/or practical (e.g. for elderly people and patients). A dynamic neuromuscular gait model capable of generating rich human-like locomotion behaviors at kinematic, kinetic and muscle levels can significantly reduce the HIL optimization time by optimizing the parameters first in the simulation and then transferring it to the hardware setup. Therefore in this project, considering the complexity of human neuromuscular control, we propose to develop a deep reinforcement learning (deep-RL) based framework capable of generating rich individual specific human walking behaviors. By simulating musculoskeletal locomotion dynamics, we expect superior predictive capabilities at three levels: (1) individual steady and non-steady gait, (2) response dynamics to unexpected perturbations, and (3) gait assistance dynamics. Here, the learned neural network of the model represents (in a schematic way) the spinal cord neural circuitry mapping sensory inputs to muscle stimulations. We plan to use the individual human rich gait data for our model to learn the walking behaviors across all the three aspects listed above. The quality of the learned gait model will be evaluated in perturbed gait scenarios and assistive gait scenarios using a leg exoskeleton. The proposed deep-RL based framework will not only improve the understanding of human neuromuscular gait control but also aid in developing AI-based gait controllers which could then be transferred to the hardware system with minimal online optimization.

肌肉骨骼疾病是导致人类残疾的主要原因之一。在德国等许多人口老龄化的国家，此类残疾的风险正在增加。辅助设备(如外骨骼、假肢等)可以促进人类的运动功能和改善步行性能(例如平衡、代谢成本等)。研究表明，采用人在环(HIL)优化方法，外骨骼具有特定于受试者的控制参数，可以改善人的行走性能。然而，HIL方法在步态多样性(例如不同的步行速度)和优化时间(例如在跑步机上连续步行一小时)方面受到限制。使用类似于HIL的方法寻找最佳参数所需的大量时间可能是不可行和/或不实际的(例如对于老年人和病人)。动态神经肌肉步态模型能够在运动学、运动学和肌肉层面产生丰富的类似人类的运动行为，通过首先在仿真中优化参数，然后将其转移到硬件设置，可以显著减少HIL优化时间。因此，在本项目中，考虑到人类神经肌肉控制的复杂性，我们提出了一种基于深度强化学习(Depth-RL)的框架，能够生成丰富的个人特定的人类行走行为。通过模拟肌肉骨骼的运动动力学，我们期望在三个水平上具有更好的预测能力：(1)个人稳定和非稳定步态，(2)对意外扰动的反应动力学，以及(3)步态辅助动力学。在这里，模型的学习神经网络表示(以示意图的方式)将感觉输入映射到肌肉刺激的脊髓神经回路。我们计划使用人类个体丰富的步态数据来学习上述三个方面的步行行为。学习的步态模型的质量将在扰动步态场景和使用腿部外骨骼的辅助步态场景中进行评估。所提出的基于深层RL的框架不仅将提高对人类神经肌肉步态控制的理解，而且有助于开发基于AI的步态控制器，然后这些控制器可以通过最小的在线优化转移到硬件系统中。