REPLUG: Retrieval-Augmented Black-Box Language Models
REPLUG: Retrieval-Augmented Black-Box Language Models
复制标题
DOI：
10.48550/arxiv.2301.12652
复制
发表时间：
2023-01
期刊：
影响因子：
--
通讯作者：
Weijia Shi;Sewon Min;Michihiro Yasunaga;Minjoon Seo;Rich James;M. Lewis;Luke Zettlemoyer;Wen-tau Yih
中科院分区：
文献类型：
--
作者：
Weijia Shi;Sewon Min;Michihiro Yasunaga;Minjoon Seo;Rich James;M. Lewis;Luke Zettlemoyer;Wen-tau Yih
我们介绍Replug，这是一种检索功能的语言建模框架，将语言模型（LM）视为黑匣子，并使用可调的检索模型将其增强。与以前的检索型LMS不同，该LMS使用特殊的跨注意机制训练语言模型来编码所检索的文本，因此，Replug只需将已检索的文档预录到冷冻的Black-Box LM的输入即可。这种简单的设计可以轻松地应用于任何现有的语言模型。此外，我们证明LM可以用来监督检索模型，然后可以找到有助于LM做出更好预测的文档。我们的实验表明，通过调整后的猎犬重新予以重复，可以显着提高GPT-3（175b）对语言建模的性能，并在5次MMLU上的Codex在5.1％上的性能提高了6.3％。代码在github.com/SWJ0419/replug上公开发布。
We introduce REPLUG, a retrieval-augmented language modeling framework that treats the language model (LM) as a black box and augments it with a tuneable retrieval model. Unlike prior retrieval-augmented LMs that train language models with special cross-attention mechanisms to encode the retrieved text, REPLUG simply prepends retrieved documents to the input for the frozen black-box LM. This simple design can be easily applied to any existing language models. Furthermore, we show that the LM can be used to supervise the retrieval model, which can then find documents that help the LM make better predictions. Our experiments demonstrate that REPLUG with the tuned retriever significantly improves the performance of GPT-3 (175B) on language modeling by 6.3%, as well as the performance of Codex on five-shot MMLU by 5.1%. Code is publicly released at github.com/swj0419/REPLUG.