Knowledge Augmentation and Task Planning in Large Language Models for Dexterous Grasping
Knowledge Augmentation and Task Planning in Large Language Models for Dexterous Grasping
复制标题
DOI：
10.1109/humanoids57100.2023.10375176
复制
发表时间：
2023-12
期刊：
影响因子：
--
通讯作者：
Hui Li;Dang M. Tran;Xinyu Zhang;Hongsheng He
中科院分区：
文献类型：
--
作者：
Hui Li;Dang M. Tran;Xinyu Zhang;Hongsheng He
灵巧的抓取能力是类人机器人与物理环境有效交互的关键能力。人类通过目标感知、物体识别、特征估计和直觉推理等一系列高级认知过程来实现灵巧的抓取。这些过程共同促进了对物体的理解和适当的抓取策略的产生。然而，目前的研究主要集中在建立大的对象数据集来估计对象的特征，以及采用学习和规划的方法来进行任务部署，对灵巧抓取的认知方面的探索有限，尤其是直觉的作用。本文通过研究灵巧抓取的认知过程来弥补这一研究空白，并提出了一种基于认知的抓取系统。所提出的系统整合了各种认知过程，以实现灵巧的抓取。它使用具有常识的大型语言模型收集对象信息并估计丢失的细节。基于补充的信息，系统学习合适的抓取策略并直观地指导它们的执行。用拟人机械手进行的真实世界实验证明了该系统的性能。通过利用认知过程和利用大型语言模型的能力，该方法增强了对对象的理解，生成了有效的抓取策略，并为抓取策略的执行提供了指导。
Dexterous grasping is a critical ability for humanoid robots to interact efficiently with the physical environment. Human beings achieve dexterous grasping through a series of high level cognitive processes including target perception, object recognition, feature estimation, and intuitive reasoning. These processes cooperatively contribute to object understanding and the generation of appropriate grasping strategies. However, the current research focuses on establishing large object datasets to estimate object features and employing learning and planning approaches for task deployment, the exploration of the cognitive aspect of dexterous grasping is limited, especially the role of intuition. This paper addresses this research gap by investigating the cognitive processes in dexterous grasping and presents a cognition based grasping system. The proposed system integrates various cognitive processes to enable dexterous grasping. It gathers object information and estimates missing details using a large language model with common sense. Based on the complemented information, the system learns suitable grasp strategies and intuitively guides their execution. Real-world experiments with a anthropomorphic robot hand demonstrated the performance of the proposed system. By leveraging cognitive processes and utilizing the capabilities of a large language model, The proposed method enhances object understanding, generates effective grasping strategies, and provides guidance for the execution of the grasping strategies.