MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text
MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text
复制标题
MuRAG：用于图像和文本开放式问答的多模态检索增强生成器
DOI：
10.48550/arxiv.2210.02928
复制
发表时间：
2022
期刊：
影响因子：
--
通讯作者：
William W. Cohen
中科院分区：
文献类型：
--
作者：
Wenhu Chen;Hexiang Hu;Xi Chen;Pat Verga;William W. Cohen
虽然语言模型在其参数中隐式存储了大量的世界知识，但即使非常大的模型也常常无法对有关罕见实体和事件的信息进行编码，同时会产生巨大的计算成本。最近，诸如 REALM、RAG 和 RETRO 之类的检索增强模型通过利用外部非参数索引将世界知识融入到语言生成中，并在模型大小受限的情况下展示了令人印象深刻的性能。然而，这些方法仅限于检索文本知识，忽略了图像等其他形式中无处不在的知识——其中大部分包含任何文本未涵盖的信息。为了解决这个限制，我们提出了第一个多模态检索增强变压器（MuRAG），它访问外部非参数多模态存储器来增强语言生成。 MuRAG 使用联合对比和生成损失，使用大规模图像文本和纯文本语料库的混合进行预训练。我们对两个不同的数据集进行实验，这两个数据集需要对图像和文本进行检索和推理来回答给定的查询：WebQA 和 MultimodalQA。我们的结果表明，MuRAG 实现了最先进的准确性，在两个数据集上以及在干扰项和全维基设置下绝对优于现有模型 10-20%。
While language Models store a massive amount of world knowledge implicitly in their parameters, even very large models often fail to encode information about rare entities and events, while incurring huge computational costs. Recently, retrieval-augmented models, such as REALM, RAG, and RETRO, have incorporated world knowledge into language generation by leveraging an external non-parametric index and have demonstrated impressive performance with constrained model sizes. However, these methods are restricted to retrieving only textual knowledge, neglecting the ubiquitous amount of knowledge in other modalities like images – much of which contains information not covered by any text. To address this limitation, we propose the first Multimodal Retrieval-Augmented Transformer (MuRAG), which accesses an external non-parametric multimodal memory to augment language generation. MuRAG is pre-trained with a mixture of large-scale image-text and text-only corpora using a joint contrastive and generative loss. We perform experiments on two different datasets that require retrieving and reasoning over both images and text to answer a given query: WebQA, and MultimodalQA. Our results show that MuRAG achieves state-of-the-art accuracy, outperforming existing models by 10-20% absolute on both datasets and under both distractor and full-wiki settings.