BERT Busters: Outlier Dimensions that Disrupt Transformers
BERT Busters: Outlier Dimensions that Disrupt Transformers
复制标题
DOI：
10.18653/v1/2021.findings-acl.300
复制
发表时间：
2021-05
期刊：
影响因子：
--
通讯作者：
Olga Kovaleva;Saurabh Kulshreshtha;Anna Rogers;Anna Rumshisky
中科院分区：
文献类型：
--
作者：
Olga Kovaleva;Saurabh Kulshreshtha;Anna Rogers;Anna Rumshisky
多项研究表明，Transformers对修剪非常健壮。与这种公认的观点相反，我们证明了预先训练的Transformer编码器对于删除层输出中的极少数特征（<模型权重的0.0001%）来说是令人惊讶的脆弱。在BERT和其他预先训练的编码器转换器的情况下，受影响的组件是LayerNorm中的缩放因子和偏差。离群值是在预训练早期出现的高量级归一化参数，并且在整个模型中始终显示在相同的维度位置。我们发现，禁用它们显着降低传销损失和下游任务的性能。在几个BERT系列模型和其他流行的预训练Transformer架构（包括BART，XLNet和ELECTRA）中观察到这种效果;我们在GPT-2中也显示了类似的效果。
Multiple studies have shown that Transformers are remarkably robust to pruning. Contrary to this received wisdom, we demonstrate that pre-trained Transformer encoders are surprisingly fragile to the removal of a very small number of features in the layer outputs (<0.0001% of model weights). In case of BERT and other pre-trained encoder Transformers, the affected component is the scaling factors and biases in the LayerNorm. The outliers are high-magnitude normalization parameters that emerge early in pre-training and show up consistently in the same dimensional position throughout the model. We show that disabling them significantly degrades both the MLM loss and the downstream task performance. This effect is observed across several BERT-family models and other popular pre-trained Transformer architectures, including BART, XLNet and ELECTRA; we also show a similar effect in GPT-2.