Accelerating attention through gradient-based learned runtime pruning
Accelerating attention through gradient-based learned runtime pruning
复制标题
DOI：
10.1145/3470496.3527423
复制
发表时间：
2022-04
期刊：
影响因子：
--
通讯作者：
Zheng Li;Soroush Ghodrati;A. Yazdanbakhsh;H. Esmaeilzadeh;Mingu Kang
中科院分区：
文献类型：
--
作者：
Zheng Li;Soroush Ghodrati;A. Yazdanbakhsh;H. Esmaeilzadeh;Mingu Kang
自我注意力是各种基于transformer的自然语言处理模型的最新准确性的关键推动因素。这种注意力机制计算句子中每个单词相对于其他单词的相关性得分。通常，只有一小部分单词与关注的单词高度相关，这仅在运行时确定。因此，由于注意力分数低，大量的计算是无关紧要的，并且可以潜在地被修剪。主要的挑战是找到分数的阈值，低于该阈值，随后的计算将是无关紧要的。虽然这样的阈值是离散的，本文制定其搜索通过一个软微分正则化集成到损失函数的训练。该公式依赖于反向传播训练，以同时分析性地共同优化阈值和权重，在准确性和计算修剪之间取得形式上的最佳平衡。为了最好地利用这一数学创新，我们设计了一个位串行架构，被称为LeOPard，Transformer语言模型与位级提前终止微架构机制。我们在MemN 2N、BERT、ALBERT、GPT-2和Vision Transformer模型的43个后端任务中评估我们的设计。布局后的结果表明，平均而言，LeOPard分别产生1.9倍和3.9倍的加速和能量降低，同时保持平均精度几乎不变（< 0.2%的退化）。
Self-attention is a key enabler of state-of-art accuracy for various transformer-based Natural Language Processing models. This attention mechanism calculates a correlation score for each word with respect to the other words in a sentence. Commonly, only a small subset of words highly correlates with the word under attention, which is only determined at runtime. As such, a significant amount of computation is inconsequential due to low attention scores and can potentially be pruned. The main challenge is finding the threshold for the scores below which subsequent computation will be inconsequential. Although such a threshold is discrete, this paper formulates its search through a soft differentiable regularizer integrated into the loss function of the training. This formulation piggy backs on the back-propagation training to analytically co-optimize the threshold and the weights simultaneously, striking a formally optimal balance between accuracy and computation pruning. To best utilize this mathematical innovation, we devise a bit-serial architecture, dubbed LeOPArd, for transformer language models with bit-level early termination microarchitectural mechanism. We evaluate our design across 43 back-end tasks for MemN2N, BERT, ALBERT, GPT-2, and Vision transformer models. Post-layout results show that, on average, LeOPArd yields 1.9×and 3.9×speedup and energy reduction, respectively, while keeping the average accuracy virtually intact (< 0.2% degradation).