Algorithm-hardware Co-design of Attention Mechanism on FPGA Devices
Algorithm-hardware Co-design of Attention Mechanism on FPGA Devices
复制标题
FPGA器件上注意力机制的算法-硬件协同设计
DOI：
10.1145/3477002
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Jingtong Hu
中科院分区：
文献类型：
--
作者：
Xinyi Zhang;Yawen Wu;Peipei Zhou;Xulong Tang;Jingtong Hu
多头自注意（注意机制）由于其在特征提取和序列数据分析方面的优势，已被应用于机器翻译、语言建模和图像处理等多种领域。这得益于注意力机制背后的大量参数和复杂的模型架构。为了在资源受限的设备上有效地部署注意力机制，现有的工作提出通过构建定制的较小模型或压缩大的标准模型来减小模型大小。定制的较小模型通常针对特定任务进行优化，并且需要在模型参数探索方面进行努力。模型压缩在不损害模型架构鲁棒性的情况下减小了模型大小，这可以有效地应用于不同的任务。模型中的压缩权重通常是规则形状的（例如矩形），但尺寸大小不同（例如矩形的高度和宽度不同）。这种压缩注意力机制可以有效地部署在CPU/GPU平台上，因为它们的存储器和计算资源可以根据需求灵活地分配。然而，对于现场可编程门阵列（FPGA），数据缓冲器分配和计算内核在运行时是固定的，以实现最大的能量效率。压缩后，权重小得多，大小不同，这导致FPGA片上缓冲区的利用效率低下。此外，不同的权重高度和宽度可能导致FPGA计算内核执行效率低下。由于注意力机制中的权值数量庞大，在FPGA上为每个压缩权值构建唯一的缓冲区和计算核是不可行的。在这项工作中，我们共同考虑压缩缓冲区分配的影响和所需的计算内核在注意力机制压缩。提出了一种新的具有内存占用感知的结构剪枝方法，并设计了相应的FPGA加速器。实验结果表明，我们的工作可以压缩Transformer（一个基于注意力机制的模型）的95倍。该加速器能够充分利用FPGA资源，在ZCU 102 FPGA上处理稀疏注意机制，运行时吞吐量性能达到1.87 Tops。
Multi-head self-attention (attention mechanism) has been employed in a variety of fields such as machine translation, language modeling, and image processing due to its superiority in feature extraction and sequential data analysis. This is benefited from a large number of parameters and sophisticated model architecture behind the attention mechanism. To efficiently deploy attention mechanism on resource-constrained devices, existing works propose to reduce the model size by building a customized smaller model or compressing a big standard model. A customized smaller model is usually optimized for the specific task and needs effort in model parameters exploration. Model compression reduces model size without hurting the model architecture robustness, which can be efficiently applied to different tasks. The compressed weights in the model are usually regularly shaped (e.g. rectangle) but the dimension sizes vary (e.g. differs in rectangle height and width). Such compressed attention mechanism can be efficiently deployed on CPU/GPU platforms as their memory and computing resources can be flexibly assigned with demand. However, for Field Programmable Gate Arrays (FPGAs), the data buffer allocation and computing kernel are fixed at run time to achieve maximum energy efficiency. After compression, weights are much smaller and different in size, which leads to inefficient utilization of FPGA on-chip buffer. Moreover, the different weight heights and widths may lead to inefficient FPGA computing kernel execution. Due to the large number of weights in the attention mechanism, building a unique buffer and computing kernel for each compressed weight on FPGA is not feasible. In this work, we jointly consider the compression impact on buffer allocation and the required computing kernel during the attention mechanism compressing. A novel structural pruning method with memory footprint awareness is proposed and the associated accelerator on FPGA is designed. The experimental results show that our work can compress Transformer (an attention mechanism based model) by 95x. The developed accelerator can fully utilize the FPGA resource, processing the sparse attention mechanism with the run-time throughput performance of 1.87 Tops in ZCU102 FPGA.
DOI：
10.1109/tcad.2020.2986127
发表时间：
2020-12-01
影响因子：
2.9
作者：
Jiang, Weiwen;Yang, Lei;Hu, Jingtong
通讯作者：
Hu, Jingtong