TIDEE: Tidying Up Novel Rooms using Visuo-Semantic Commonsense Priors
TIDEE: Tidying Up Novel Rooms using Visuo-Semantic Commonsense Priors
复制标题
TIDEE：使用视觉语义常识先验整理新房间
DOI：
10.48550/arxiv.2207.10761
复制
发表时间：
2022
期刊：
影响因子：
--
通讯作者：
Katerina Fragkiadaki
中科院分区：
文献类型：
--
作者：
Gabriel H. Sarch;Zhaoyuan Fang;Adam W. Harley;Paul Schydlo;M. Tarr;Saurabh Gupta;Katerina Fragkiadaki
我们介绍了潮汐，体现代理，整理了一个混乱的场景的基础上学到的常识对象的位置和房间安排先验。TIDEE探索家庭环境，检测出它们的自然位置的对象，为它们推断出合理的对象上下文，在当前场景中定位这些上下文，并重新定位对象。常识先验被编码在三个模块中：i）视觉语义检测器，其检测不适当的对象; ii）对象和空间关系的关联神经图存储器，其提出用于对象重新定位的合理语义容器和表面;以及iii）视觉搜索网络，其引导代理的探索以有效地定位当前场景中的感兴趣的容器以重新定位对象。我们在AI 2 THOR模拟环境中测试TIDEE整理杂乱无章的场景。TIDEE直接从像素和原始深度输入执行任务，而无需事先观察同一个房间，仅依赖于从一组单独的训练室学习的先验知识。对由此产生的房间重组的人类评估显示，TIDEE优于不使用一个或多个常识先验的模型的消融版本。在一个相关的房间重排基准，允许代理查看目标状态之前，重新安排，我们的模型的简化版本显着优于一个最高性能的方法由一个大的利润率。代码和数据可在项目网站上获得：https://tidee-agent.github.io/。
We introduce TIDEE, an embodied agent that tidies up a disordered scene based on learned commonsense object placement and room arrangement priors. TIDEE explores a home environment, detects objects that are out of their natural place, infers plausible object contexts for them, localizes such contexts in the current scene, and repositions the objects. Commonsense priors are encoded in three modules: i) visuo-semantic detectors that detect out-of-place objects, ii) an associative neural graph memory of objects and spatial relations that proposes plausible semantic receptacles and surfaces for object repositions, and iii) a visual search network that guides the agent's exploration for efficiently localizing the receptacle-of-interest in the current scene to reposition the object. We test TIDEE on tidying up disorganized scenes in the AI2THOR simulation environment. TIDEE carries out the task directly from pixel and raw depth input without ever having observed the same room beforehand, relying only on priors learned from a separate set of training houses. Human evaluations on the resulting room reorganizations show TIDEE outperforms ablative versions of the model that do not use one or more of the commonsense priors. On a related room rearrangement benchmark that allows the agent to view the goal state prior to rearrangement, a simplified version of our model significantly outperforms a top-performing method by a large margin. Code and data are available at the project website: https://tidee-agent.github.io/.
DOI：
--
发表时间：
2021-07
期刊：
ArXiv
影响因子：
--
作者：
Valts Blukis;Chris Paxton;D. Fox;Animesh Garg;Yoav Artzi
通讯作者：
Valts Blukis;Chris Paxton;D. Fox;Animesh Garg;Yoav Artzi
通过观看 YouTube 视频进行语义视觉导航
DOI：
--
发表时间：
2020
期刊：
2020
影响因子：
--
作者：
Chang, Matthew;Gupta, Arjun;Gupta, Saurabh
通讯作者：
Gupta, Saurabh