NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM
NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM
复制标题
DOI：
--
复制
发表时间：
2021-10
期刊：
影响因子：
--
通讯作者：
Connor Holmes;Minjia Zhang;Yuxiong He;Bo Wu
中科院分区：
文献类型：
--
作者：
Connor Holmes;Minjia Zhang;Yuxiong He;Bo Wu
自然语言处理（NLP）最近通过使用巨大的预训练Transformer网络取得了成功。然而，这些模型通常包含数亿甚至数十亿个参数，由于延迟限制，给在线部署带来了挑战。最近，硬件制造商已经为NxM稀疏性引入了专用硬件，以提供非结构化修剪的灵活性和结构化方法的运行时效率。NXM稀疏性允许从密集表示中的N个连续组中任意选择M个参数来保留。然而，由于预训练模型的复杂性极高，标准的稀疏微调技术往往无法很好地推广到数据资源有限的下游任务。为了以原则性的方式解决这个问题，我们引入了一个新的学习框架，称为NxMTransformer，以诱导预训练语言模型的NxM半结构化稀疏性，用于自然语言理解，以获得更好的性能。特别是，我们建议制定的NxM稀疏作为一个约束优化问题，并使用交替方向乘法（ADMM）优化下游任务，同时考虑到底层的硬件约束。ADMM将NxM稀疏化问题分解为两个可以依次解决的子问题，生成稀疏化的Transformer网络，实现高精度，同时能够在新发布的硬件上有效执行。我们将我们的方法应用于广泛的NLP任务，我们提出的方法能够在GLUE得分中实现比当前实践高1.7分的准确性。此外，我们对我们的方法进行了详细的分析，并阐明了ADMM如何影响下游任务的微调精度。最后，我们说明了如何NxMTransformer实现知识蒸馏的性能改进。
Natural Language Processing (NLP) has recently achieved success by using huge pre-trained Transformer networks. However, these models often contain hundreds of millions or even billions of parameters, bringing challenges to online deployment due to latency constraints. Recently, hardware manufacturers have introduced dedicated hardware for NxM sparsity to provide the flexibility of unstructured pruning with the runtime efficiency of structured approaches. NxM sparsity permits arbitrarily selecting M parameters to retain from a contiguous group of N in the dense representation. However, due to the extremely high complexity of pre-trained models, the standard sparse fine-tuning techniques often fail to generalize well on downstream tasks, which have limited data resources. To address such an issue in a principled manner, we introduce a new learning framework, called NxMTransformer, to induce NxM semi-structured sparsity on pretrained language models for natural language understanding to obtain better performance. In particular, we propose to formulate the NxM sparsity as a constrained optimization problem and use Alternating Direction Method of Multipliers (ADMM) to optimize the downstream tasks while taking the underlying hardware constraints into consideration. ADMM decomposes the NxM sparsification problem into two sub-problems that can be solved sequentially, generating sparsified Transformer networks that achieve high accuracy while being able to effectively execute on newly released hardware. We apply our approach to a wide range of NLP tasks, and our proposed method is able to achieve 1.7 points higher accuracy in GLUE score than current practices. Moreover, we perform detailed analysis on our approach and shed light on how ADMM affects fine-tuning accuracy for downstream tasks. Finally, we illustrate how NxMTransformer achieves performance improvement with knowledge distillation.