Re2G: Retrieve, Rerank, Generate
Re2G: Retrieve, Rerank, Generate
复制标题
DOI：
10.48550/arxiv.2207.06300
复制
发表时间：
2022-07
期刊：
影响因子：
--
通讯作者：
Michael R. Glass;Gaetano Rossiello;Md. Faisal Mahbub Chowdhury;Ankita Rajaram Naik;Pengshan Cai;A. Gliozzo
中科院分区：
文献类型：
--
作者：
Michael R. Glass;Gaetano Rossiello;Md. Faisal Mahbub Chowdhury;Ankita Rajaram Naik;Pengshan Cai;A. Gliozzo
正如GPT - 3和T5所展示的那样，随着参数空间变得越来越大，Transformer的能力也在增强。然而，对于需要大量知识的任务，非参数记忆允许模型在计算成本和GPU内存需求呈次线性增长的情况下大幅增长。最近的一些模型，如RAG和REALM，已经将检索引入到条件生成中。这些模型结合了从段落语料库中进行的神经初始检索。我们在这一系列研究的基础上，提出了Re2G，它将神经初始检索和重新排序结合到基于BART的序列到序列生成中。我们的重新排序方法还允许合并来自分数不可比的来源的检索结果，从而实现BM25和神经初始检索的集成。为了对我们的系统进行端到端的训练，我们引入了一种新的知识蒸馏变体，仅使用目标序列输出的基本事实来训练初始检索、重新排序器和生成器。我们在四个不同的任务中发现了巨大的收益：零样本槽填充、问答、事实核查和对话，在KILT排行榜上比之前的最先进技术相对提高了9%到34%。我们将我们的代码开源。
As demonstrated by GPT-3 and T5, transformers grow in capability as parameter spaces become larger and larger. However, for tasks that require a large amount of knowledge, non-parametric memory allows models to grow dramatically with a sub-linear increase in computational cost and GPU memory requirements. Recent models such as RAG and REALM have introduced retrieval into conditional generation. These models incorporate neural initial retrieval from a corpus of passages. We build on this line of research, proposing Re2G, which combines both neural initial retrieval and reranking into a BART-based sequence-to-sequence generation. Our reranking approach also permits merging retrieval results from sources with incomparable scores, enabling an ensemble of BM25 and neural initial retrieval. To train our system end-to-end, we introduce a novel variation of knowledge distillation to train the initial retrieval, reranker and generation using only ground truth on the target sequence output. We find large gains in four diverse tasks: zero-shot slot filling, question answering, fact checking and dialog, with relative gains of 9% to 34% over the previous state-of-the-art on the KILT leaderboard. We make our code available as open source.