Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube
Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube
复制标题
DOI：
10.15607/rss.2022.xviii.023
复制
发表时间：
2022-02
期刊：
影响因子：
--
通讯作者：
Aravind Sivakumar;Kenneth Shaw;Deepak Pathak
中科院分区：
文献类型：
--
作者：
Aravind Sivakumar;Kenneth Shaw;Deepak Pathak
我们建立了一个系统，使任何人都能控制机器人的手和手臂，只需用自己的手示范动作。机器人通过一个单一的RGB摄像头观察人类操作员，并实时模仿他们的动作。人的手和机器人的手在形状、大小和关节结构上都不同，从一个未校准的相机进行这种转换是一个高度缺乏约束的问题。此外，重新定位的轨迹必须有效地执行物理机器人上的任务，这就要求它们在时间上是平滑的，并且没有自我碰撞。我们的关键观点是，虽然收集成对的人机通信数据非常昂贵，但互联网包含大量丰富多样的人手视频。我们利用这些数据来训练一个系统，该系统可以理解人类的手，并将人类视频流重新定位为机器人的手臂轨迹，该轨迹平滑、快速、安全，并且在语义上与指导演示相似。我们证明，它使以前未经训练的人能够远程操作机器人进行各种灵巧的操作任务。我们的低成本、无手套、无标记的远程操作系统使机器人教学更容易实现，我们希望它能帮助机器人学习在现实世界中自主行动。视频请访问https://robotic-telekinesis.github.io/
We build a system that enables any human to control a robot hand and arm, simply by demonstrating motions with their own hand. The robot observes the human operator via a single RGB camera and imitates their actions in real-time. Human hands and robot hands differ in shape, size, and joint structure, and performing this translation from a single uncalibrated camera is a highly underconstrained problem. Moreover, the retargeted trajectories must effectively execute tasks on a physical robot, which requires them to be temporally smooth and free of self-collisions. Our key insight is that while paired human-robot correspondence data is expensive to collect, the internet contains a massive corpus of rich and diverse human hand videos. We leverage this data to train a system that understands human hands and retargets a human video stream into a robot hand-arm trajectory that is smooth, swift, safe, and semantically similar to the guiding demonstration. We demonstrate that it enables previously untrained people to teleoperate a robot on various dexterous manipulation tasks. Our low-cost, glove-free, marker-free remote teleoperation system makes robot teaching more accessible and we hope that it can aid robots in learning to act autonomously in the real world. Videos at https://robotic-telekinesis.github.io/