Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models
Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models
复制标题
DOI：
10.48550/arxiv.2203.03131
复制
发表时间：
2022-03
期刊：
影响因子：
--
通讯作者：
Shengnan An;Yifei Li;Zeqi Lin;Qian Liu;Bei Chen-;Qiang Fu;Weizhu Chen;Nanning Zheng;Jian-Guang Lou
中科院分区：
文献类型：
--
作者：
Shengnan An;Yifei Li;Zeqi Lin;Qian Liu;Bei Chen-;Qiang Fu;Weizhu Chen;Nanning Zheng;Jian-Guang Lou
最近，自动调优范式引起了极大的关注。通过仅使用冻结的预训练语言模型（PLM）调整连续提示，快速调整朝着部署共享的冻结PLM以服务于众多下游任务迈出了一步。尽管自动调优在某些自然语言理解（NLU）任务上表现出良好的性能，但其在自然语言生成（NLG）任务上的有效性仍有待探索。在本文中，我们认为，阻碍NLG任务的快速调整发展的因素之一是不熟悉的输入（即，输入在语言上不同于预训练语料库）。例如，我们的初步探索揭示了一个很大的性能差距时，不熟悉的输入经常出现在NLG任务的微调和微调。这促使我们提出输入调整，微调连续提示和输入表示，导致一个更有效的方式来适应不熟悉的输入冻结PLM。我们提出的输入调整概念简单，经验强大。七个NLG任务的实验结果表明，输入调整明显且一致地优于提示调整。此外，在其中三个任务上，输入调整可以实现与微调相当甚至更好的性能。
Recently the prompt-tuning paradigm has attracted significant attention. By only tuning continuous prompts with a frozen pre-trained language model (PLM), prompt-tuning takes a step towards deploying a shared frozen PLM to serve numerous downstream tasks. Although prompt-tuning shows good performance on certain natural language understanding (NLU) tasks, its effectiveness on natural language generation (NLG) tasks is still under-explored. In this paper, we argue that one of the factors hindering the development of prompt-tuning on NLG tasks is the unfamiliar inputs (i.e., inputs are linguistically different from the pretraining corpus). For example, our preliminary exploration reveals a large performance gap between prompt-tuning and fine-tuning when unfamiliar inputs occur frequently in NLG tasks. This motivates us to propose input-tuning, which fine-tunes both the continuous prompts and the input representations, leading to a more effective way to adapt unfamiliar inputs to frozen PLMs. Our proposed input-tuning is conceptually simple and empirically powerful. Experimental results on seven NLG tasks demonstrate that input-tuning is significantly and consistently better than prompt-tuning. Furthermore, on three of these tasks, input-tuning can achieve a comparable or even better performance than fine-tuning.