Improving the understanding of neuromuscular gait control using deep reinforcement learning

肌肉骨骼疾病是导致人类残疾的主要原因之一。在许多像德国这样的人口老龄化国家，这种残疾的风险正在增加。辅助装置（如外骨骼、假肢等）可以促进人体运动功能，改善步行性能（如平衡、代谢成本等）。研究表明，采用人在环（human-in-the-loop， HIL）优化方法，外骨骼具有受试者特定的控制参数，可以改善人类的行走性能。然而，HIL方法在步态多样性（例如不同的步行速度）和优化时间（例如在跑步机上连续步行一小时）方面受到限制。使用类hil方法寻找最佳参数所需的大量时间可能不可行和/或不实用（例如，对于老年人和患者）。动态神经肌肉步态模型能够在运动学、动力学和肌肉水平上产生丰富的类人运动行为，通过首先在仿真中优化参数，然后将其转移到硬件设置，可以显着减少HIL优化时间。因此，在这个项目中，考虑到人类神经肌肉控制的复杂性，我们建议开发一个基于深度强化学习（deep- rl）的框架，能够生成丰富的个体特定的人类行走行为。通过模拟肌肉骨骼运动动力学，我们期望在三个层面上具有卓越的预测能力：(1)个体稳定和非稳定步态，(2)对意外扰动的响应动力学，以及(3)步态辅助动力学。这里，模型的学习神经网络（以示意图的方式）表示将感觉输入映射到肌肉刺激的脊髓神经回路。我们计划在我们的模型中使用个人丰富的步态数据来学习上述三个方面的行走行为。学习步态模型的质量将在摄动步态场景和使用腿部外骨骼的辅助步态场景中进行评估。提出的基于深度强化学习的框架不仅可以提高对人类神经肌肉步态控制的理解，还有助于开发基于人工智能的步态控制器，然后通过最小的在线优化将其转移到硬件系统。

Musculoskeletal disorders are one of the leading causes of human disability. The risk of such disabilities is increasing in many countries with an aging population like Germany. Assistive devices (e.g. exoskeletons, prosthesis, etc.) can facilitate human locomotor function and improve walking performance (e.g. balance, metabolic cost, etc.). It has been shown that human walking performance can be improved by the exoskeleton with subject specific control parameters using the human-in-the-loop (HIL) optimization approach. However, the HIL approach is limited in terms of gait diversity (e.g. different walking speeds) and optimization time (e.g. one hour of continuous walking on a treadmill). The large amount of time required to find the optimal parameters using a HIL-like method might not be feasible and/or practical (e.g. for elderly people and patients). A dynamic neuromuscular gait model capable of generating rich human-like locomotion behaviors at kinematic, kinetic and muscle levels can significantly reduce the HIL optimization time by optimizing the parameters first in the simulation and then transferring it to the hardware setup. Therefore in this project, considering the complexity of human neuromuscular control, we propose to develop a deep reinforcement learning (deep-RL) based framework capable of generating rich individual specific human walking behaviors. By simulating musculoskeletal locomotion dynamics, we expect superior predictive capabilities at three levels: (1) individual steady and non-steady gait, (2) response dynamics to unexpected perturbations, and (3) gait assistance dynamics. Here, the learned neural network of the model represents (in a schematic way) the spinal cord neural circuitry mapping sensory inputs to muscle stimulations. We plan to use the individual human rich gait data for our model to learn the walking behaviors across all the three aspects listed above. The quality of the learned gait model will be evaluated in perturbed gait scenarios and assistive gait scenarios using a leg exoskeleton. The proposed deep-RL based framework will not only improve the understanding of human neuromuscular gait control but also aid in developing AI-based gait controllers which could then be transferred to the hardware system with minimal online optimization.