Chasing Sparsity in Vision Transformers: An End-to-End Exploration
Chasing Sparsity in Vision Transformers: An End-to-End Exploration
复制标题
DOI：
--
复制
发表时间：
2021-06
期刊：
影响因子：
--
通讯作者：
Tianlong Chen;Yu Cheng;Zhe Gan;Lu Yuan;Lei Zhang;Zhangyang Wang
中科院分区：
文献类型：
--
作者：
Tianlong Chen;Yu Cheng;Zhe Gan;Lu Yuan;Lei Zhang;Zhangyang Wang
视觉变压器（ViTs）最近得到了爆炸性的普及，但它们巨大的模型尺寸和培训成本仍然令人望而生畏。传统的培训后修剪通常会招致更高的培训预算。相比之下，本文的目的是在不牺牲可实现的准确性的前提下，减少训练内存开销和推理复杂性。我们首次对采用统一的方法“从端到端”集成vit的稀疏性进行了全面的探索。具体来说，我们不是训练完整的vit，而是动态地提取和训练稀疏的子网络，同时坚持一个固定的小参数预算。我们的方法在整个训练过程中共同优化模型参数并探索连通性，最终得到一个稀疏网络作为最终输出。该方法从非结构化稀疏无缝扩展到结构化稀疏，后者通过考虑引导vit内部自关注头的修剪和生长。我们进一步共同探索数据和架构稀疏性，通过插入一个新的可学习令牌选择器来自适应地确定当前最重要的补丁，从而获得额外的效率提升。在具有不同ViT主干的ImageNet上的大量结果验证了我们的建议的有效性，这些建议大大降低了计算成本，并且几乎没有损害泛化。也许最令人惊讶的是，我们发现所提出的稀疏（协同）训练有时可以提高而不是降低ViT的准确性，使稀疏性成为诱人的“免费午餐”。例如，我们对（数据，架构）进行了（5%,50%）稀疏化的DeiT-Small，提高了0.28%的top-1精度，同时获得了49.32%的FLOPs和4.40%的运行时间节省。我们的代码可在https://github.com/VITA-Group/SViTE上获得。
Vision transformers (ViTs) have recently received explosive popularity, but their enormous model sizes and training costs remain daunting. Conventional post-training pruning often incurs higher training budgets. In contrast, this paper aims to trim down both the training memory overhead and the inference complexity, without sacrificing the achievable accuracy. We carry out the first-of-its-kind comprehensive exploration, on taking a unified approach of integrating sparsity in ViTs"from end to end". Specifically, instead of training full ViTs, we dynamically extract and train sparse subnetworks, while sticking to a fixed small parameter budget. Our approach jointly optimizes model parameters and explores connectivity throughout training, ending up with one sparse network as the final output. The approach is seamlessly extended from unstructured to structured sparsity, the latter by considering to guide the prune-and-grow of self-attention heads inside ViTs. We further co-explore data and architecture sparsity for additional efficiency gains by plugging in a novel learnable token selector to adaptively determine the currently most vital patches. Extensive results on ImageNet with diverse ViT backbones validate the effectiveness of our proposals which obtain significantly reduced computational cost and almost unimpaired generalization. Perhaps most surprisingly, we find that the proposed sparse (co-)training can sometimes improve the ViT accuracy rather than compromising it, making sparsity a tantalizing"free lunch". For example, our sparsified DeiT-Small at (5%, 50%) sparsity for (data, architecture), improves 0.28% top-1 accuracy, and meanwhile enjoys 49.32% FLOPs and 4.40% running time savings. Our codes are available at https://github.com/VITA-Group/SViTE.