Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
复制标题
DOI：
10.48550/arxiv.2212.10509
复制
发表时间：
2022-12
期刊：
影响因子：
--
通讯作者：
H. Trivedi;Niranjan Balasubramanian;Tushar Khot;Ashish Sabharwal
中科院分区：
文献类型：
--
作者：
H. Trivedi;Niranjan Balasubramanian;Tushar Khot;Ashish Sabharwal
基于促进的大型语言模型（LLMS）在生成自然语言推理步骤或思想链（COT）以进行多步骤问题（QA）方面非常强大。但是，当必要的知识无法在其参数范围内最新或不采用最新时，他们会挣扎。在使用该问题从外部知识来源检索相关文本有助于LLMS时，我们观察到这种单步检索方法不足以用于多步质量质量质量质量标准。在这里，检索的内容取决于已经得出的内容，这反过来可能取决于先前检索的内容。为了解决这个问题，我们提出了Ircot，这是一种多步质量保证的新方法，它与COT中的步骤（句子）结合了检索，并用COT指导检索，然后依次使用检索结果来改善COT。使用与GPT3的IRCOT可以在四个数据集上使用IRCOT可改善检索（最多21点）以及下游QA（最多15分）：HotPotQA，2Wikimultihopqa，Musique和IIRC。我们观察到在分布（OOD）设置以及诸如Flan-T5-Large之类的较小型号的情况下，也有类似的实质性收益，而无需额外的培训。 IRCOT减少了模型幻觉，从而导致了更准确的COT推理。
Prompting-based large language models (LLMs) are surprisingly powerful at generating natural language reasoning steps or Chains-of-Thoughts (CoT) for multi-step question answering (QA). They struggle, however, when the necessary knowledge is either unavailable to the LLM or not up-to-date within its parameters. While using the question to retrieve relevant text from an external knowledge source helps LLMs, we observe that this one-step retrieve-and-read approach is insufficient for multi-step QA. Here, what to retrieve depends on what has already been derived, which in turn may depend on what was previously retrieved. To address this, we propose IRCoT, a new approach for multi-step QA that interleaves retrieval with steps (sentences) in a CoT, guiding the retrieval with CoT and in turn using retrieved results to improve CoT. Using IRCoT with GPT3 substantially improves retrieval (up to 21 points) as well as downstream QA (up to 15 points) on four datasets: HotpotQA, 2WikiMultihopQA, MuSiQue, and IIRC. We observe similar substantial gains in out-of-distribution (OOD) settings as well as with much smaller models such as Flan-T5-large without additional training. IRCoT reduces model hallucination, resulting in factually more accurate CoT reasoning.