Toward Grounded Commonsense Reasoning
Toward Grounded Commonsense Reasoning
复制标题
DOI：
--
复制
发表时间：
2023-06
期刊：
影响因子：
--
通讯作者：
Minae Kwon;Hengyuan Hu;Vivek Myers;Siddharth Karamcheti;A. Dragan;Dorsa Sadigh
中科院分区：
文献类型：
--
作者：
Minae Kwon;Hengyuan Hu;Vivek Myers;Siddharth Karamcheti;A. Dragan;Dorsa Sadigh
想象一下，一个机器人的任务是用一辆精心制作的乐高跑车整理桌子。人类可能认识到，作为“整理”的一部分，拆卸跑车并将其收起来是不合适的。“一个机器人怎么能得出这样的结论呢？尽管大型语言模型（LLM）最近已被用于实现常识推理，但将这种推理建立在真实的世界中一直具有挑战性。为了在真实的世界中进行推理，机器人必须超越被动地查询LLM，并主动从环境中收集做出正确决策所需的信息。例如，在检测到有被遮挡的汽车之后，机器人可能需要主动感知汽车，以知道它是由乐高制成的高级模型汽车还是由蹒跚学步的孩子建造的玩具汽车。我们提出了一种方法，利用LLM和视觉语言模型（VLM）来帮助机器人主动感知其环境，以执行接地常识推理。为了大规模评估我们的框架，我们发布了MessySurfaces数据集，其中包含需要清理的70个真实表面的图像。我们还说明了我们的方法与机器人2精心设计的表面。我们发现，与不使用主动感知的基线相比，MessySurfaces基准平均提高了12.9%，机器人实验平均提高了15%。我们的方法的数据集、代码和视频可以在https://minaek.github.io/grounded_commonsense_reasoning上找到。
Consider a robot tasked with tidying a desk with a meticulously constructed Lego sports car. A human may recognize that it is not appropriate to disassemble the sports car and put it away as part of the"tidying."How can a robot reach that conclusion? Although large language models (LLMs) have recently been used to enable commonsense reasoning, grounding this reasoning in the real world has been challenging. To reason in the real world, robots must go beyond passively querying LLMs and actively gather information from the environment that is required to make the right decision. For instance, after detecting that there is an occluded car, the robot may need to actively perceive the car to know whether it is an advanced model car made out of Legos or a toy car built by a toddler. We propose an approach that leverages an LLM and vision language model (VLM) to help a robot actively perceive its environment to perform grounded commonsense reasoning. To evaluate our framework at scale, we release the MessySurfaces dataset which contains images of 70 real-world surfaces that need to be cleaned. We additionally illustrate our approach with a robot on 2 carefully designed surfaces. We find an average 12.9% improvement on the MessySurfaces benchmark and an average 15% improvement on the robot experiments over baselines that do not use active perception. The dataset, code, and videos of our approach can be found at https://minaek.github.io/grounded_commonsense_reasoning.