Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
复制标题
DOI：
10.1145/3560815
复制
发表时间：
2023-09-01
影响因子：
16.6
通讯作者：
Neubig, Graham
中科院分区：
文献类型：
--
作者：
Liu, Pengfei;Yuan, Weizhe;Neubig, Graham
本文调查并组织了研究研究的研究，以新的自然语言处理范式进行，我们将其称为“基于及时的学习”。与传统的监督学习不同，该学习训练模型以获取输入X并预测输出Y为P（Y | X），基于及时的学习是基于直接建模文本概率的语言模型。要使用这些模型执行预测任务，使用模板将原始输入X修改为具有一些未填充插槽的文本字符串提示x'，然后使用语言模型来概率地填充未填充的信息以获得最终的字符串（ x）在上限上可以得出最终输出y。由于多种原因，该框架具有强大的功能：它允许语言模型在大量的原始文本中进行预训练，并且通过定义新提示功能，该模型能够执行少数几个甚至零弹射学习，适应很少或没有标记的数据的新方案。在本文中，我们介绍了这种有前途的范式的基础知识，描述了一组统一的数学符号，可以涵盖各种各样的现有作品，并沿着几个维度组织现有工作，例如，选择预训练的语言模型，提示，提示和调整策略。为了使有兴趣的初学者更容易获得该领域，我们不仅对现有作品进行系统的审查，并对基于迅速的概念进行了高度结构化的类型，而且还发布了其他资源，例如，NLPEDIA网站NLPEDIA-PROTER，包括不断更新的调查和纸质列表。
This article surveys and organizes research works in a new paradigm in natural language processing, which we dub "prompt-based learning." Unlike traditional supervised learning, which trains a model to take in an input x and predict an output y as P (y|x), prompt-based learning is based on language models that model the probability of text directly. To use these models to perform prediction tasks, the original input x is modified using a template into a textual string prompt x' that has some unfilled slots, and then the language model is used to probabilistically fill the unfilled information to obtain a final string (x) over cap, from which the final output y can be derived. This framework is powerful and attractive for a number of reasons: It allows the language model to be pre-trained on massive amounts of raw text, and by defining a new prompting function the model is able to perform few-shot or even zero-shot learning, adapting to new scenarios with few or no labeled data. In this article, we introduce the basics of this promising paradigm, describe a unified set of mathematical notations that can cover a wide variety of existing work, and organize existing work along several dimensions, e.g., the choice of pre-trained language models, prompts, and tuning strategies. To make the field more accessible to interested beginners, we not only make a systematic review of existing works and a highly structured typology of prompt-based concepts but also release other resources, e.g., a website NLPedia-Pretrain including constantly updated survey and paperlist.