The Lottery Ticket Hypothesis for Pre-trained BERT Networks
The Lottery Ticket Hypothesis for Pre-trained BERT Networks
复制标题
DOI：
--
复制
发表时间：
2020-07
期刊：
影响因子：
--
通讯作者：
Tianlong Chen;Jonathan Frankle;Shiyu Chang;Sijia Liu;Yang Zhang;Zhangyang Wang;Michael Carbin
中科院分区：
文献类型：
--
作者：
Tianlong Chen;Jonathan Frankle;Shiyu Chang;Sijia Liu;Yang Zhang;Zhangyang Wang;Michael Carbin
在自然语言处理（NLP）中，像BERT这样庞大的预训练模型已经成为一系列下游任务的标准起点，类似的趋势也出现在深度学习的其他领域。与此同时，关于彩票假设的研究表明，NLP和计算机视觉模型包含较小的匹配子网络，能够单独训练到完全准确并转移到其他任务。在这项工作中，我们联合收割机这些观察，以评估是否存在这样的可训练，可转移的子网络在预先训练的BERT模型。对于一系列下游任务，我们确实找到了40%到90%稀疏度的匹配子网络。我们在（预先训练的）初始化时发现这些子网络，这与之前的NLP研究不同，因为它们只在经过一定量的训练后才会出现。在掩蔽语言建模任务（用于预训练模型的相同任务）中发现的子网络普遍转移;在其他任务中发现的子网络以有限的方式转移。随着大规模预训练成为深度学习中越来越重要的范式，我们的研究结果表明，主要的彩票观察结果在这种情况下仍然是相关的。此https URL上提供的代码。
In natural language processing (NLP), enormous pre-trained models like BERT have become the standard starting point for training on a range of downstream tasks, and similar trends are emerging in other areas of deep learning. In parallel, work on the lottery ticket hypothesis has shown that models for NLP and computer vision contain smaller matching subnetworks capable of training in isolation to full accuracy and transferring to other tasks. In this work, we combine these observations to assess whether such trainable, transferrable subnetworks exist in pre-trained BERT models. For a range of downstream tasks, we indeed find matching subnetworks at 40% to 90% sparsity. We find these subnetworks at (pre-trained) initialization, a deviation from prior NLP research where they emerge only after some amount of training. Subnetworks found on the masked language modeling task (the same task used to pre-train the model) transfer universally; those found on other tasks transfer in a limited fashion if at all. As large-scale pre-training becomes an increasingly central paradigm in deep learning, our results demonstrate that the main lottery ticket observations remain relevant in this context. Codes available at this https URL.