AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning
复制标题
DOI：
10.48550/arxiv.2210.17451
复制
发表时间：
2022-05
期刊：
影响因子：
--
通讯作者：
Yaqing Wang;Subhabrata Mukherjee;Xiaodong Liu;Jing Gao;Jianfeng Gao
中科院分区：
文献类型：
--
作者：
Yaqing Wang;Subhabrata Mukherjee;Xiaodong Liu;Jing Gao;Jianfeng Gao
针对下游任务的大型预训练语言模型（PLM）的标准微调需要更新数亿到数十亿个参数，并为每个任务存储PLM权重的大型副本，从而导致存储、共享和服务模型的成本增加。为了解决这个问题，引入了参数高效微调（PEFT）技术，其中将小型可训练组件注入PLM并在微调期间更新。我们提出AdaMix作为一种通用的PEFT方法，它可以调整每个Transformer层中引入的适应模块的混合-给定选择的底层PEFT方法，同时保持大多数PLM权重冻结。例如，AdaMix可以利用像Houlsby这样的适配器的混合或像LoRA这样的低秩分解矩阵的混合来提高下游任务的性能，而不是完全监督和少量NLU和NLG任务的相应PEFT方法。此外，我们设计AdaMix，使其匹配相同的计算成本和可调参数的数量作为底层PEFT方法。通过仅调整0.1-0.2%的PLM参数，我们表明AdaMix在NLU和NLG任务中的性能优于SOTA参数高效微调和全模型微调。
Standard fine-tuning of large pre-trained language models (PLMs) for downstream tasks requires updating hundreds of millions to billions of parameters, and storing a large copy of the PLM weights for every task resulting in increased cost for storing, sharing and serving the models. To address this, parameter-efficient fine-tuning (PEFT) techniques were introduced where small trainable components are injected in the PLM and updated during fine-tuning. We propose AdaMix as a general PEFT method that tunes a mixture of adaptation modules – given the underlying PEFT method of choice – introduced in each Transformer layer while keeping most of the PLM weights frozen. For instance, AdaMix can leverage a mixture of adapters like Houlsby or a mixture of low rank decomposition matrices like LoRA to improve downstream task performance over the corresponding PEFT methods for fully supervised and few-shot NLU and NLG tasks. Further, we design AdaMix such that it matches the same computational cost and the number of tunable parameters as the underlying PEFT method. By only tuning 0.1-0.2% of PLM parameters, we show that AdaMix outperforms SOTA parameter-efficient fine-tuning and full model fine-tuning for both NLU and NLG tasks.