CAREER: Hierarchical Reinforcement Learning Framework for Safe Dynamic Bipedal Locomotion

双足机器人的拟人设计使这些机器具有独特的优势，可以在具有挑战性的地形中航行（例如，自然不平坦的地面、山丘和楼梯），在受限的环境中操作（例如，为人类操作员设计的房屋或仓库中的狭窄垂直空间），并以自然的方式与其他人交互。然而，由于对运动控制器的基本机制缺乏了解，在双足机器人中实现安全和动态行为的技术仍然具有挑战性。这种理解也将有助于改善辅助设备，如下肢外骨骼，这可能有助于恢复由于中风或其他运动障碍而失去的活动能力。该学院早期职业发展（CAREER）项目旨在通过一种新型的基于学习的反馈运动控制框架，显着推进双足机器人和下肢外骨骼技术，特别关注在现实世界中通过实验实现安全和动态的双足运动。受人类如何分层学习复杂任务的启发，该项目将通过物理启发的分层结构对运动控制进行重大创新，以实现安全的双足运动。此外，我们将在协调的教育和外展计划中利用双足机器人的先天吸引力，让代表性不足的各个级别的学生参与STEM教育和研究项目。该项目的成功完成有可能加速双足机器人在工业和公共卫生领域的实际应用，并提高STEM人才库的多样性。双足机器人具有固有的不稳定性，并且由许多自由度组成。尽管目前在机器人操作和移动的机器人方面取得了成就，但典型的强化学习（RL）算法在用于双足机器人时容易失败并且难以扩展。机器人将在没有适当控制的情况下摔倒，导致非常稀疏和不连续的奖励，导致RL算法无法收敛。双足机器人的高维性也增加了经典“扁平”RL算法的搜索空间，导致采样效率低下和冗长（可能不成功）的训练过程。RL在双足机器人上的许多现有应用不尊重机器人的物理限制，因此不能在机器人硬件上实现。因此，本研究将通过追求以下四个研究目标来应对这些科学挑战：（G1）开发分层学习结构，其使得机器人能够通过经由不同级别的技能的时间抽象来降低任务复杂度而有效地探索高维行为空间，（G2）设计概率安全的“安全过滤器”，以经由控制屏障功能来确保和引导安全策略学习，（G3）通过以分层方式模仿受物理启发的模板模型的引导策略探索来提高学习效率，最后（G4）弥合现实世界部署的“模拟到真实的”差距。由此产生的框架将实验证明与3D双足机器人（Digit）在现实世界中的设置和下肢外骨骼（ATALANTE）在非脊髓损伤健康人类受试者的临床前试验中。该项目得到机器人项目跨董事会基础研究的支持，由工程局（ENG）和计算机与信息科学与工程局（CISE）共同管理和资助该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The anthropomorphic design of bipedal robots equips these machines with unique advantages navigating challenging terrains (e.g., natural uneven grounds, hills, and stairs), operating in restricted environments (e.g., narrow vertical spaces in house or warehouse designed for human operators), and interacting with other humans in a natural manner. However, the technological realization of safe and dynamic behaviors in bipedal robots remains challenging due to the fundamental lack of understanding of underlying mechanisms of locomotion controllers. Such understanding will also help improve assistive devices such as lower-limb exoskeletons, which could help restore mobility lost due to stroke or other movement disorders. This Faculty Early Career Development (CAREER) project aims to significantly advance the technology of bipedal robots and lower-limb exoskeletons through a novel learning-based feedback motion control framework, with a specific focus on experimentally realizing safe and dynamic bipedal locomotion in real-world settings. Inspired by how humans learn complex tasks hierarchically, this project will make major innovations to motion control for safe bipedal locomotion through a physics-inspired hierarchical structure. Moreover, we will leverage the innate appeal of bipedal robots in our coordinated education and outreach plans to engage underrepresented students of various levels in STEM education and research programs. The successful completion of this project has the potential to accelerate real-world applications of bipedal robots in industry and public health and improve the diversity in the STEM talent pool.Bipedal robots are inherently unstable and consist of many degrees of freedom. Despite current achievements in robot manipulation and mobile robots, typical reinforcement learning (RL) algorithms are prone to fail and are difficult to scale when used for bipedal robots. The robot will fall without proper control, resulting in very sparse and discontinuous rewards, causing the RL algorithms not to converge. The high dimensionality of bipedal robots also increases the search space of the classic “flat” RL algorithms, leading to sampling inefficiency and a lengthy (potentially unsuccessful) training process. Many existing applications of RL on bipedal robots do not respect the physical limitations of the robot, and consequently, cannot be implemented on robot hardware. This research will therefore address these scientific challenges by pursuing the following four research goals: (G1) develop a hierarchical learning structure that enables the robot to efficiently explore the high-dimensional behavior space by reducing the task complexity through the temporal abstraction of skills at different levels, (G2) design a probabilistically safe “safety filter” to ensure and guide safe policy learning via control barrier functions, (G3) improve learning efficiency through guided policy exploration that imitates physics-inspired template models in a layered fashion, and finally (G4) bridge the “sim-to-real” gap for real-world deployments. The resulting framework will be experimentally demonstrated with a 3D bipedal robot (Digit) in real-world settings and a lower-limb exoskeleton (ATALANTE) in pre-clinical trials with non-Spinal-Cord-Injury healthy human subjects.This project is supported by the cross-directorate Foundational Research in Robotics program, jointly managed and funded by the Directorates for Engineering (ENG) and Computer and Information Science and Engineering (CISE).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.