ViTALiTy: Unifying Low-rank and Sparse Approximation for Vision Transformer Acceleration with a Linear Taylor Attention
ViTALiTy: Unifying Low-rank and Sparse Approximation for Vision Transformer Acceleration with a Linear Taylor Attention
复制标题
DOI：
10.1109/hpca56546.2023.10071081
复制
发表时间：
2022-11
期刊：
影响因子：
--
通讯作者：
Jyotikrishna Dass;Shang Wu;Huihong Shi;Chaojian Li;Zhifan Ye;Zhongfeng Wang;Yingyan Lin
中科院分区：
文献类型：
--
作者：
Jyotikrishna Dass;Shang Wu;Huihong Shi;Chaojian Li;Zhifan Ye;Zhongfeng Wang;Yingyan Lin
Vision Transformer（ViT）已成为各种计算机视觉应用中卷积神经网络的竞争替代方案。具体来说，ViTs的多头注意力层可以在整个图像中全局嵌入信息。然而，计算和存储这样的注意力矩阵会导致二次成本依赖于补丁的数量，限制了其可实现的效率和可扩展性，并禁止更广泛的现实世界中的ViT应用资源受限的设备。稀疏注意力已被证明是提高NLP模型硬件加速效率的一个有前途的方向。然而，一个系统的对应方法仍然缺乏加速ViT模型。为了缩小上述差距，我们提出了一个首创的算法硬件协同设计的框架，称为VITALITY，用于提高ViTs的推理效率。与NLP的基于稀疏性的Transformer加速器不同，VITALITY统一了ViTs中注意力的低秩和稀疏成分。在算法层面，我们通过一阶Taylor注意力近似点积softmax操作，行均值中心作为低秩分量，以线性化注意力块的成本，并通过结合基于稀疏的正则化进一步提高准确性。在硬件层面，我们开发了一个专用的加速器，以更好地利用VITALITY的线性泰勒注意力所产生的工作负载和流水线，该注意力只需要执行低秩组件，以进一步提高硬件效率。广泛的实验和消融研究证实，VITALITY提供了提高的端到端效率（例如，相对于最先进的解决方案，在相当的精度下，速度提高3倍，能效提高3倍）。我们在https://github.com/GATECH-EIC/ViTaLiTy上提供代码
Vision Transformer (ViT) has emerged as a competitive alternative to convolutional neural networks for various computer vision applications. Specifically, ViTs’ multi-head attention layers make it possible to embed information globally across the overall image. Nevertheless, computing and storing such attention matrices incurs a quadratic cost dependency on the number of patches, limiting its achievable efficiency and scalability and prohibiting more extensive real-world ViT applications on resource-constrained devices. Sparse attention has been shown to be a promising direction for improving hardware acceleration efficiency for NLP models. However, a systematic counterpart approach is still missing for accelerating ViT models. To close the above gap, we propose a first-of-its-kind algorithm-hardware codesigned framework, dubbed VITALITY, for boosting the inference efficiency of ViTs. Unlike sparsity-based Transformer accelerators for NLP, VITALITY unifies both low-rank and sparse components of the attention in ViTs. At the algorithm level, we approximate the dot-product softmax operation via first-order Taylor attention with row-mean centering as the low-rank component to linearize the cost of attention blocks and further boost the accuracy by incorporating a sparsity-based regularization. At the hardware level, we develop a dedicated accelerator to better leverage the resulting workload and pipeline from VITALITY’s linear Taylor attention which requires the execution of only the low-rank component, to further boost the hardware efficiency. Extensive experiments and ablation studies validate that VITALITY offers boosted end-to-end efficiency (e.g., 3× faster and 3× energy-efficient) under comparable accuracy, with respect to the state-of-the-art solution. We make the codes available on https://github.com/GATECH-EIC/ViTaLiTy