ReTransformer: ReRAM-based Processing-in-Memory Architecture for Transformer Acceleration
ReTransformer: ReRAM-based Processing-in-Memory Architecture for Transformer Acceleration
复制标题
DOI：
10.1145/3400302.3415640
复制
发表时间：
2020-11
期刊：
影响因子：
--
通讯作者：
Xiaoxuan Yang;Bonan Yan;H. Li;Yiran Chen
中科院分区：
文献类型：
--
作者：
Xiaoxuan Yang;Bonan Yan;H. Li;Yiran Chen
Transformer已成为一种流行的深度神经网络（DNN）模型，用于神经语言处理（NLP）应用，并在神经机器翻译、实体识别等方面表现出优异的性能。然而，自回归解码器的尺度点积注意机制在推理过程中带来了性能瓶颈。Transformer也是计算和内存密集型的，需要硬件加速解决方案。尽管研究人员已经成功地应用了基于reram的内存处理（PIM）来加速卷积神经网络（cnn）和递归神经网络（rnn），但Transformer中缩放点积注意力的独特计算过程使得这些设计难以直接应用。此外，如何处理矩阵-矩阵乘法（MatMul）中的中间结果，以及如何在Transformer的更细粒度上设计管道，都是尚未解决的问题。在这项工作中，我们提出了ReTransformer -一个基于reram的用于Transformer加速的PIM架构。ReTransformer不仅可以使用基于reram的PIM加速Transformer的缩放点积注意力，而且可以使用所提出的矩阵分解技术避免编写中间结果，从而消除一些数据依赖性。此外，我们提出了一种新的子矩阵管道设计，用于多头自关注。实验结果表明，与GPU和Pipelayer相比，ReTransformer的计算效率分别提高了23.21倍和3.25倍。相应的总功率分别降低1086倍和2.82倍。
Transformer has emerged as a popular deep neural network (DNN) model for Neural Language Processing (NLP) applications and demonstrated excellent performance in neural machine translation, entity recognition, etc. However, its scaled dot-product attention mechanism in auto-regressive decoder brings a performance bottleneck during inference. Transformer is also computationally and memory intensive and demands for a hardware acceleration solution. Although researchers have successfully applied ReRAM-based Processing-in-Memory (PIM) to accelerate convolutional neural networks (CNNs) and recurrent neural networks (RNNs), the unique computation process of the scaled dot-product attention in Transformer makes it difficult to directly apply these designs. Besides, how to handle intermediate results in Matrix-matrix Multiplication (MatMul) and how to design a pipeline at a finer granularity of Transformer remain unsolved. In this work, we propose ReTransformer - a ReRAM-based PIM architecture for Transformer acceleration. ReTransformer can not only accelerate the scaled dot-product attention of Transformer using ReRAM-based PIM but also eliminate some data dependency by avoiding writing the intermediate results using the proposed matrix decomposition technique. Moreover, we propose a new sub-matrix pipeline design for multi-head self-attention. Experimental results show that compared to GPU and Pipelayer, ReTransformer improves computing efficiency by 23.21× and 3.25×, respectively. The corresponding overall power is reduced by 1086× and 2.82×, respectively.