A length adaptive algorithm-hardware co-design of transformer on FPGA through sparse attention and dynamic pipelining
A length adaptive algorithm-hardware co-design of transformer on FPGA through sparse attention and dynamic pipelining
复制标题
DOI：
10.1145/3489517.3530585
复制
发表时间：
2022-07
期刊：
影响因子：
--
通讯作者：
Hongwu Peng;Shaoyi Huang;Shiyang Chen;Bingbing Li;Tong Geng;Ang Li;Weiwen Jiang;Wujie Wen;J. Bi;Hang Liu;Caiwen Ding
中科院分区：
文献类型：
--
作者：
Hongwu Peng;Shaoyi Huang;Shiyang Chen;Bingbing Li;Tong Geng;Ang Li;Weiwen Jiang;Wujie Wen;J. Bi;Hang Liu;Caiwen Ding
自2018年以来，变形金刚被认为是最重要的深度学习模型之一，部分原因是它建立了最先进的记录（SOTA）记录，并有可能取代现有的深神经网络（DNN）。变压器模型是一个广泛认识的障碍。阵列（FPGA）并提出了一个连贯的序列长度，尤其是变压器加速度，我们开发了一个硬件友好的稀疏注意操作员和一个长度感知的硬件资源调度算法。基于注意力的模型，可减轻芯片的内存流量。与CPU和GPU实施相比，具有非常小的精度损失，并且具有80.2×和2.6倍的加速，并且比通过Cublas Gemm优化的最先进的GPU加速器高4倍。
Transformers are considered one of the most important deep learning models since 2018, in part because it establishes state-of-the-art (SOTA) records and could potentially replace existing Deep Neural Networks (DNNs). Despite the remarkable triumphs, the prolonged turnaround time of Transformer models is a widely recognized roadblock. The variety of sequence lengths imposes additional computing overhead where inputs need to be zero-padded to the maximum sentence length in the batch to accommodate the parallel computing platforms. This paper targets the field-programmable gate array (FPGA) and proposes a coherent sequence length adaptive algorithm-hardware co-design for Transformer acceleration. Particularly, we develop a hardware-friendly sparse attention operator and a length-aware hardware resource scheduling algorithm. The proposed sparse attention operator brings the complexity of attention-based models down to linear complexity and alleviates the off-chip memory traffic. The proposed length-aware resource hardware scheduling algorithm dynamically allocates the hardware resources to fill up the pipeline slots and eliminates bubbles for NLP tasks. Experiments show that our design has very small accuracy loss and has 80.2 × and 2.6 × speedup compared to CPU and GPU implementation, and 4 × higher energy efficiency than state-of-the-art GPU accelerator optimized via CUBLAS GEMM.