M2T2: Multi-Task Masked Transformer for Object-centric Pick and Place
M2T2: Multi-Task Masked Transformer for Object-centric Pick and Place
复制标题
DOI：
10.48550/arxiv.2311.00926
复制
发表时间：
2023-11
期刊：
影响因子：
--
通讯作者：
Wentao Yuan;Adithyavairavan Murali;Arsalan Mousavian;Dieter Fox
中科院分区：
文献类型：
--
作者：
Wentao Yuan;Adithyavairavan Murali;Arsalan Mousavian;Dieter Fox
随着大型语言模型和大规模机器人数据集的出现，在对象操作的高层决策方面取得了巨大的进步。这些通用模型能够使用语言命令解释复杂的任务，但由于无法使用低级操作原语，它们通常难以泛化到分布之外的对象。相比之下，现有的特定于任务的模型擅长于对未知对象的低级操作，但仅适用于单一类型的操作。为了弥补这一差距，我们提出了M2T2，这是一个单一的模型，它提供了不同类型的低级操作，可以在混乱场景中的任意对象上健壮地工作。M2T2是一个变形模型，它可以对接触点进行推理，并在给定场景的原始点云的情况下，为不同的动作模式预测有效的抓手姿势。在具有128K场景的大规模合成数据集上进行训练，M2T2在真实机器人上实现了零镜头sim2Real传输，在整体性能上比采用最先进任务特定模型的基线系统高出约19%，在具有挑战性的场景中性能高出37.5%，其中对象需要重新定位以实现无碰撞放置。M2T2还在RLBitch语言条件化任务的子集上实现了最先进的结果。机器人在现实世界和模拟世界中对看不见的物体进行实验的视频可以在我们的项目网站https://m2-t2.github.io.上找到
With the advent of large language models and large-scale robotic datasets, there has been tremendous progress in high-level decision-making for object manipulation. These generic models are able to interpret complex tasks using language commands, but they often have difficulties generalizing to out-of-distribution objects due to the inability of low-level action primitives. In contrast, existing task-specific models excel in low-level manipulation of unknown objects, but only work for a single type of action. To bridge this gap, we present M2T2, a single model that supplies different types of low-level actions that work robustly on arbitrary objects in cluttered scenes. M2T2 is a transformer model which reasons about contact points and predicts valid gripper poses for different action modes given a raw point cloud of the scene. Trained on a large-scale synthetic dataset with 128K scenes, M2T2 achieves zero-shot sim2real transfer on the real robot, outperforming the baseline system with state-of-the-art task-specific models by about 19% in overall performance and 37.5% in challenging scenes where the object needs to be re-oriented for collision-free placement. M2T2 also achieves state-of-the-art results on a subset of language conditioned tasks in RLBench. Videos of robot experiments on unseen objects in both real world and simulation are available on our project website https://m2-t2.github.io.