RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning-猫眼课题宝

DOI：

10.48550/arxiv.2205.12548

发表时间：

2022-05

期刊：

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Mingkai Deng;Jianyu Wang;Cheng-Ping Hsieh;Yihan Wang-;Han Guo;Tianmin Shu;Meng Song;E. Xing;Zhiting Hu

在使大型预训练语言模型（LM）能够执行各种NLP任务方面，特别是在只有很少下游数据的情况下，人工智能已经取得了令人印象深刻的成功。然而，自动为每个任务找到最佳提示是具有挑战性的。大多数现有的工作诉诸于调整 * 软 * 提示（例如，嵌入），这些都缺乏可解释性、跨LM的可重用性以及当梯度不可访问时的适用性。另一方面，* 离散 * 提示难以优化，并且通常通过“枚举”（例如，释义）-然后选择”的策略，不系统地探索提示空间。提出了一种基于强化学习的高效离散提示优化方法RLPrompt。RLPrompt制定了一个参数有效的策略网络，该网络在带奖励的训练后生成优化的离散提示。为了利用来自大型LM环境的复杂和随机的奖励信号，我们引入了有效的奖励稳定，大大提高了训练效率。RLPrompt可灵活地应用于不同类型的LM，例如掩码（例如，BERT）和从左到右模型（例如，GPT），用于分类和生成任务。关于少次分类和无监督文本风格迁移的实验表明，与各种现有的微调或提示方法相比，该方法具有优越的上级性能。有趣的是，所得到的优化提示通常是不合语法的胡言乱语文本;并且令人惊讶的是，这些胡言乱语提示可以在不同的LM之间转移以保持显著的性能，这表明LM提示可能不遵循人类语言模式。

Prompting has shown impressive success in enabling large pre-trained language models (LMs) to perform diverse NLP tasks, especially with only few downstream data. Automatically finding the optimal prompt for each task, however, is challenging. Most existing work resorts to tuning *soft* prompts (e.g., embeddings) which fall short of interpretability, reusability across LMs, and applicability when gradients are not accessible. *Discrete* prompts, on the other hand, are difficult to optimize, and are often created by “enumeration (e.g., paraphrasing)-then-selection” heuristics that do not explore the prompt space systematically. This paper proposes RLPrompt, an efficient discrete prompt optimization approach with reinforcement learning (RL). RLPrompt formulates a parameter-efficient policy network that generates the optimized discrete prompt after training with reward. To harness the complex and stochastic reward signals from the large LM environment, we incorporate effective reward stabilization that substantially enhances training efficiency. RLPrompt is flexibly applicable to different types of LMs, such as masked (e.g., BERT) and left-to-right models (e.g., GPTs), for both classification and generation tasks. Experiments on few-shot classification and unsupervised text style transfer show superior performance over a wide range of existing fine-tuning or prompting methods. Interestingly, the resulting optimized prompts are often ungrammatical gibberish text; and surprisingly, those gibberish prompts are transferrable between different LMs to retain significant performance, indicating that LM prompting may not follow human language patterns.

RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning

文献摘要

相似文献