Complementary Explanations for Effective In-Context Learning
Complementary Explanations for Effective In-Context Learning
复制标题
DOI：
10.48550/arxiv.2211.13892
复制
发表时间：
2022-11
期刊：
影响因子：
--
通讯作者：
Xi Ye;Srini Iyer;Asli Celikyilmaz;Ves Stoyanov;Greg Durrett;Ramakanth Pasunuru
中科院分区：
文献类型：
--
作者：
Xi Ye;Srini Iyer;Asli Celikyilmaz;Ves Stoyanov;Greg Durrett;Ramakanth Pasunuru
大型语言模型（LLM）在从提示中的解释中学习方面表现出了非凡的能力，但对这些解释的确切功能或为什么它们有效的理解有限。这项工作的目的是更好地了解的机制，解释用于在上下文学习。我们首先研究了两个不同因素对提示性能的影响：计算轨迹（解决方案的分解方式）和用于表达提示的自然语言。通过对三个受控任务的扰动解释，我们发现这两个因素都有助于解释的有效性。我们进一步研究如何形成最有效的解释解决一个给定的测试查询集。我们发现，LLM可以受益于解释集的互补性：不同的范例所表现出的不同推理技能可以带来更好的表现。因此，我们提出了一种基于最大边缘相关性的样本选择方法，用于构建相关且互补的样本集，该方法成功地提高了多个LLM上三个真实任务的上下文学习性能。
Large language models (LLMs) have exhibited remarkable capabilities in learning from explanations in prompts, but there has been limited understanding of exactly how these explanations function or why they are effective. This work aims to better understand the mechanisms by which explanations are used for in-context learning. We first study the impact of two different factors on the performance of prompts with explanations: the computation trace (the way the solution is decomposed) and the natural language used to express the prompt. By perturbing explanations on three controlled tasks, we show that both factors contribute to the effectiveness of explanations. We further study how to form maximally effective sets of explanations for solving a given test query. We find that LLMs can benefit from the complementarity of the explanation set: diverse reasoning skills shown by different exemplars can lead to better performance. Therefore, we propose a maximal marginal relevance-based exemplar selection approach for constructing exemplar sets that are both relevant as well as complementary, which successfully improves the in-context learning performance across three real-world tasks on multiple LLMs.