Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization
Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization
复制标题
DOI：
10.1109/fpl57034.2022.00027
复制
发表时间：
2022-08
期刊：
影响因子：
--
通讯作者：
Z. Li;Mengshu Sun-;Alec Lu;Haoyu Ma;Geng Yuan;Yanyue Xie;Hao Tang;Yanyu Li;M. Leeser;Zhangyang Wang;Xue Lin;Zhenman Fang
中科院分区：
文献类型：
--
作者：
Z. Li;Mengshu Sun-;Alec Lu;Haoyu Ma;Geng Yuan;Yanyue Xie;Hao Tang;Yanyu Li;M. Leeser;Zhangyang Wang;Xue Lin;Zhenman Fang
视觉变压器（ViTs）的出现大大提高了计算机视觉任务的精度。然而，它们复杂的体系结构和巨大的计算/存储需求迫切需要新的硬件加速器设计方法。本文提出了一种基于混合方案量化的fpga感知ViT自动加速框架。据我们所知，这是第一个基于fpga的ViT加速框架探索模型量化。与最先进的ViT量化工作（仅采用算法方法而不使用硬件加速）相比，在相同比特宽度下，我们的量化实现了0.47%至1.36%的top - 1精度提高。与32位浮点基准FPGA加速器相比，我们的加速器在帧速率上实现了约5.6倍的改进（即56.8 FPS vs. 10.0 FPS），在ImageNet数据集上的DeiT-base精度下降了0.71%。
Vision transformers (ViTs) are emerging with significantly improved accuracy in computer vision tasks. However, their complex architecture and enormous computation/storage demand impose urgent needs for new hardware accelerator design methodology. This work proposes an FPGA-aware automatic ViT acceleration framework based on the proposed mixed-scheme quantization. To the best of our knowledge, this is the first FPGA-based ViT acceleration framework exploring model quantization. Compared with state-of-the-art ViT quantization work (algorithmic approach only without hardware acceleration), our quantization achieves 0.47% to 1.36% higher Top-l accuracy under the same bit-width. Compared with the 32-bit floating-point baseline FPGA accelerator, our accelerator achieves around 5.6x improvement on the frame rate (i.e., 56.8 FPS vs. 10.0 FPS) with 0.71% accuracy drop on ImageNet dataset for DeiT-base.