Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learning
Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learning
复制标题
DOI：
10.48550/arxiv.2210.10041
复制
发表时间：
2022-10
期刊：
影响因子：
--
通讯作者：
Shuo Xie;Jiahao Qiu;Ankita Pasad;Li Du;Qing Qu;Hongyuan Mei
中科院分区：
文献类型：
--
作者：
Shuo Xie;Jiahao Qiu;Ankita Pasad;Li Du;Qing Qu;Hongyuan Mei
在传递预训练的语言模型时，常见的方法通常将特定于任务的分类器附加到顶层，并适应所有预训练的层。我们研究是否可以做出特定于任务的选择，以适应层的哪个子集以及在哪里放置分类器。目标是在不牺牲性能的情况下减少迁移学习方法（例如微调或适配器调优）的计算成本。我们建议根据给定任务特定语料库的隐藏状态的可变性来选择层。如果一个层的隐藏状态的类内可变性相对于类间可变性较低，我们就说这个层在任务中已经“很好地专门化”了。我们的可变性度量的计算成本很低，不需要任何训练或超参数调优。它对数据不平衡和数据稀缺性具有较强的鲁棒性。在GLUE基准测试上进行的大量实验表明，根据我们的度量选择层可以产生比使用相同数量的顶层更强的性能，并且通常与整个语言模型的微调或适配器调优的性能相匹配。
While transferring a pretrained language model, common approaches conventionally attach their task-specific classifiers to the top layer and adapt all the pretrained layers. We investigate whether one could make a task-specific selection on which subset of the layers to adapt and where to place the classifier. The goal is to reduce the computation cost of transfer learning methods (e.g. fine-tuning or adapter-tuning) without sacrificing its performance. We propose to select layers based on the variability of their hidden states given a task-specific corpus. We say a layer is already"well-specialized"in a task if the within-class variability of its hidden states is low relative to the between-class variability. Our variability metric is cheap to compute and doesn't need any training or hyperparameter tuning. It is robust to data imbalance and data scarcity. Extensive experiments on the GLUE benchmark demonstrate that selecting layers based on our metric can yield significantly stronger performance than using the same number of top layers and often match the performance of fine-tuning or adapter-tuning the entire language model.