Weakly Supervised Relative Spatial Reasoning for Visual Question Answering
Weakly Supervised Relative Spatial Reasoning for Visual Question Answering
复制标题
DOI：
10.1109/iccv48922.2021.00192
复制
发表时间：
2021-09
期刊：
影响因子：
--
通讯作者：
Pratyay Banerjee;Tejas Gokhale;Yezhou Yang;Chitta Baral
中科院分区：
文献类型：
--
作者：
Pratyay Banerjee;Tejas Gokhale;Yezhou Yang;Chitta Baral
视觉和语言推理需要对物体和动作等视觉概念的感知，理解语义和语言基础，以及对两种模式之间相互作用的推理。视觉推理的一个关键方面是空间理解，这涉及到理解物体的相对位置，即隐性地学习场景的几何形状。在这项工作中，我们通过制定作为分类和回归任务的成对相对位置的预测来评估V&L模型对这种几何理解的可靠性。我们的研究结果表明，最先进的基于变压器的V&L模型缺乏足够的能力来胜任这项任务。基于此，我们设计了两个目标作为3D空间推理（SR）的代理——物体质心估计和相对位置估计，并在现有深度估计器的弱监督下训练V&L。这大大提高了“GQA”视觉问题回答挑战（在完全监督、少镜头和O.O.D设置下）的准确性，并提高了相对空间推理。代码和数据将在这里发布。
Vision-and-language (V&L) reasoning necessitates perception of visual concepts such as objects and actions, understanding semantics and language grounding, and reasoning about the interplay between the two modalities. One crucial aspect of visual reasoning is spatial understanding, which involves understanding relative locations of objects, i.e. implicitly learning the geometry of the scene. In this work, we evaluate the faithfulness of V&L models to such geometric understanding, by formulating the prediction of pair-wise relative locations of objects as a classification as well as a regression task. Our findings suggest that state-of-the-art transformer-based V&L models lack sufficient abilities to excel at this task. Motivated by this, we design two objectives as proxies for 3D spatial reasoning (SR) – object centroid estimation, and relative position estimation, and train V&L with weak supervision from off-the-shelf depth estimators. This leads to considerable improvements in accuracy for the "GQA" visual question answering challenge (in fully supervised, few-shot, and O.O.D settings) as well as improvements in relative spatial reasoning. Code and data will be released here.