IS-GGT: Iterative Scene Graph Generation with Generative Transformers
IS-GGT: Iterative Scene Graph Generation with Generative Transformers
复制标题
DOI：
10.1109/cvpr52729.2023.00609
复制
发表时间：
2022-11
期刊：
影响因子：
--
通讯作者：
Sanjoy Kundu;Sathyanarayanan N. Aakur
中科院分区：
文献类型：
--
作者：
Sanjoy Kundu;Sathyanarayanan N. Aakur
场景图通过以图形格式对实体（对象）及其空间关系进行编码，提供了场景的丰富、结构化的表示。这种表示已经证明在几个任务中是有用的，例如问答，字幕，甚至对象检测，仅举几例。当前的方法采用分类生成方法，其中场景图是通过标记场景中对象之间的所有可能边缘来生成的，这增加了该方法的计算开销。这项工作介绍了一种基于生成变换的方法来生成场景图超越链接预测。使用两个基于transformer的组件，我们首先从检测到的对象及其视觉特征中采样一个可能的场景图结构。然后，我们对采样的边缘进行谓词分类，以生成最终的场景图。这种方法使我们能够有效地生成场景图的图像与最小的推理开销。在Visual Genome数据集上的大量实验证明了该方法的有效性。在没有花里胡哨的情况下，我们在场景图生成（SGG）的不同设置中平均获得了20.7%的平均召回率（mR@100），优于最先进的SGG方法，同时为无偏见的SGG方法提供了有竞争力的性能。
Scene graphs provide a rich, structured representation of a scene by encoding the entities (objects) and their spatial relationships in a graphical format. This representation has proven useful in several tasks, such as question answering, captioning, and even object detection, to name a few. Current approaches take a generation-by-classification approach where the scene graph is generated through labeling of all possible edges between objects in a scene, which adds computational overhead to the approach. This work introduces a generative transformer-based approach to generating scene graphs beyond link prediction. Using two transformer-based components, we first sample a possible scene graph structure from detected objects and their visual features. We then perform predicate classification on the sampled edges to generate the final scene graph. This approach allows us to efficiently generate scene graphs from images with minimal inference overhead. Extensive experiments on the Visual Genome dataset demonstrate the efficiency of the proposed approach. Without bells and whistles, we obtain, on average, 20.7% mean recall (mR@100) across different settings for scene graph generation (SGG), outperforming state-of-the-art SGG approaches while offering competitive performance to unbiased SGG approaches.