ELECTRA: PRE-TRAINING TEXT ENCODERS AS DISCRIMINATORS RATHER THAN GENERATORS
ELECTRA: PRE-TRAINING TEXT ENCODERS AS DISCRIMINATORS RATHER THAN GENERATORS
复制标题
DOI：
10.48550/arxiv.2003.10555
复制
发表时间：
2020-03-23
影响因子：
4.9
通讯作者：
Manning, Christopher D.
中科院分区：
文献类型：
--
作者：
Clark, Kevin;Luong, Minh-Thang;Manning, Christopher D.
掩码语言建模 (MLM) 预训练方法（例如 BERT）通过用 [MASK] 替换一些标记来破坏输入，然后训练模型来重建原始标记。虽然它们在转移到下游 NLP 任务时会产生良好的结果，但它们通常需要大量计算才能有效。作为替代方案，我们提出了一种样本效率更高的预训练任务，称为替换标记检测。我们的方法不是屏蔽输入，而是通过用从小型生成器网络采样的合理替代品替换一些令牌来破坏输入。然后，我们不是训练一个预测损坏令牌原始身份的模型，而是训练一个判别模型来预测损坏输入中的每个令牌是否被生成器样本替换。彻底的实验表明，这种新的预训练任务比 MLM 更有效，因为该任务是在所有输入标记上定义的，而不仅仅是被屏蔽的小子集。因此，在相同的模型大小、数据和计算条件下，我们的方法学习到的上下文表示明显优于 BERT 学习到的上下文表示。对于小型型号来说，收益尤其强劲；例如，我们在一个 GPU 上训练一个模型 4 天，该模型在 GLUE 自然语言理解基准测试中的性能优于 GPT（使用多 30 倍的计算量进行训练）。我们的方法在规模上也表现良好，它的性能与 RoBERTa 和 XLNet 相当，但使用的计算量不到它们的 1/4，并且在使用相同计算量时优于它们。
Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when using the same amount of compute.