Towards Addressing the Misalignment of Object Proposal Evaluation for Vision-Language Tasks via Semantic Grounding
Towards Addressing the Misalignment of Object Proposal Evaluation for Vision-Language Tasks via Semantic Grounding
复制标题
DOI：
10.1109/wacv57701.2024.00434
复制
发表时间：
2023-09
期刊：
影响因子：
--
通讯作者：
Joshua Forster Feinglass;Yezhou Yang
中科院分区：
文献类型：
--
作者：
Joshua Forster Feinglass;Yezhou Yang
目标候选区域生成是视觉 - 语言（VL）任务（图像字幕、视觉问答等）中的一个标准预处理步骤。目前，针对VL任务生成的目标候选区域的性能是根据所有可用的标注进行评估的，我们发现这种评估方式是“错位的”——较高的分数并不一定对应下游VL任务性能的提高。我们的工作对这一现象进行了研究，并探索了语义基础在减轻其影响方面的有效性。为此，我们建议仅根据可用标注的一个子集来评估目标候选区域，该子集是通过对标注重要性分数设置阈值来选择的。通过从描述图像的文本中提取相关语义信息来量化目标标注对VL任务的重要性。我们表明，与现有技术相比，我们的方法是一致的，并且与通过图像字幕指标和人工标注选择的标注具有更好的一致性。最后，我们将场景图生成（SGG）基准中使用的当前检测器作为一个用例进行比较，这是传统目标候选区域评估技术错位的一个示例。
Object proposal generation serves as a standard preprocessing step in Vision-Language (VL) tasks (image captioning, visual question answering, etc.). The performance of object proposals generated for VL tasks is currently evaluated across all available annotations, a protocol that we show is "misaligned" - higher scores do not necessarily correspond to improved performance on downstream VL tasks. Our work serves as a study of this phenomenon and explores the effectiveness of semantic grounding to mitigate its effects. To this end, we propose evaluating object proposals against only a subset of available annotations, selected by thresholding an annotation importance score. Importance of object annotations to VL tasks is quantified by extracting relevant semantic information from text describing the image. We show that our method is consistent and demonstrates greatly improved alignment with annotations selected by image captioning metrics and human annotation when compared against existing techniques. Lastly, we compare current detectors used in the Scene Graph Generation (SGG) benchmark as a use case, which serves as an example of when traditional object proposal evaluation techniques are misaligned1.