On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers
On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers
复制标题
DOI：
10.18653/v1/2021.findings-acl.363
复制
发表时间：
2021-06
期刊：
影响因子：
--
通讯作者：
Tianchu Ji;Shraddhan Jain;M. Ferdman;Peter Milder;H. A. Schwartz;Niranjan Balasubramanian
中科院分区：
文献类型：
--
作者：
Tianchu Ji;Shraddhan Jain;M. Ferdman;Peter Milder;H. A. Schwartz;Niranjan Balasubramanian
在应用程序时（推理），NLP任务真正需要多少信息？从最近的工作中，我们知道变压器存在稀疏性，并且其计算中的浮点数可以离散为更少的值，而任务精度的损失最小。但是，这需要进行重新训练甚至创建全新的型号，两者都可能是昂贵且发射碳。专注于不需要培训的优化，我们系统地研究了必要的全部典型注意值。这为推理时间量化技术的设计提供了修剪和对数尺度映射的设计，该映射仅产生少数（例如$ 2^3 $）唯一值。在回答和情感分析的任务中，我们发现将近80％的注意值可以用最小的（$ <1.0 \％$）的相对损失来修剪到零。我们将这种修剪技术与将注意值量化为仅3位格式的情况下，而无需重新训练，从而在问题上仅通过微调的Roberta降低了0.8％的准确性。
How much information do NLP tasks really need from a transformer's attention mechanism at application-time (inference)? From recent work, we know that there is sparsity in transformers and that the floating-points within its computation can be discretized to fewer values with minimal loss to task accuracies. However, this requires retraining or even creating entirely new models, both of which can be expensive and carbon-emitting. Focused on optimizations that do not require training, we systematically study the full range of typical attention values necessary. This informs the design of an inference-time quantization technique using both pruning and log-scaled mapping which produces only a few (e.g. $2^3$) unique values. Over the tasks of question answering and sentiment analysis, we find nearly 80% of attention values can be pruned to zeros with minimal ($<1.0\%$) relative loss in accuracy. We use this pruning technique in conjunction with quantizing the attention values to only a 3-bit format, without retraining, resulting in only a 0.8% accuracy reduction on question answering with fine-tuned RoBERTa.