Pre-Training Multi-Modal Dense Retrievers for Outside-Knowledge Visual Question Answering
Pre-Training Multi-Modal Dense Retrievers for Outside-Knowledge Visual Question Answering
复制标题
DOI：
10.1145/3578337.3605137
复制
发表时间：
2023-06
期刊：
影响因子：
--
通讯作者：
Alireza Salemi;Mahta Rafiee;Hamed Zamani
中科院分区：
文献类型：
--
作者：
Alireza Salemi;Mahta Rafiee;Hamed Zamani
本文研究了一类视觉问答任务，在这类任务中，回答问题需要获取外部知识。这一类别被称为非知识视觉问答(OK-VQA)。开发OK-VQA系统的一个重要步骤是为给定的多模式查询检索相关文档。目前针对该任务的非对称密集检索模型采用多模式查询编码器和单模式文档编码器的体系结构。这样的体系结构需要大量的训练数据才能有效地执行。提出了一种用于OK-VQA任务的预训练通道检索模型的数据自动生成流水线。与当前最先进的非对称体系结构相比，建议的方法使Precision@5提高了26.9%。此外，提出的预训练方法在零镜头检索场景中表现出了良好的性能。
This paper studies a category of visual question answering tasks, in which accessing external knowledge is necessary for answering the questions. This category is called outside-knowledge visual question answering (OK-VQA). A major step in developing OK-VQA systems is to retrieve relevant documents for the given multi-modal query. Current state-of-the-art asymmetric dense retrieval model for this task uses an architecture with a multi-modal query encoder and a uni-modal document encoder. Such an architecture requires a large amount of training data for effective performance. We propose an automatic data generation pipeline for pre-training passage retrieval models for OK-VQA tasks. The proposed approach leads to 26.9% Precision@5 improvements compared to the current state-of-the-art asymmetric architecture. Additionally, the proposed pre-training approach exhibits a good ability in zero-shot retrieval scenarios.