Adaptive Inverse Transform Sampling For Efficient Vision Transformers
Adaptive Inverse Transform Sampling For Efficient Vision Transformers
复制标题
DOI：
--
复制
发表时间：
2022
期刊：
影响因子：
--
通讯作者：
Mohsen Fayyaz-;Soroush Abbasi Koohpayegani;F. Jafari;Sunando Sengupta;Hamid Reza Vaezi Joze;Eric Sommerlade
中科院分区：
文献类型：
--
作者：
Mohsen Fayyaz-;Soroush Abbasi Koohpayegani;F. Jafari;Sunando Sengupta;Hamid Reza Vaezi Joze;Eric Sommerlade
.虽然最先进的视觉Transformer模型在图像分类方面取得了令人满意的结果，但它们在计算上是昂贵的，并且需要许多GFLOP。虽然可以通过减少网络中的令牌数量来降低视觉变换器的GFLOP，但没有对所有输入图像都最佳的设置。因此，在这项工作中，我们引入了一个可微分的无参数自适应令牌采样（ATS）模块，它可以插入到任何现有的视觉Transformer架构。ATS通过对重要标记进行评分和自适应采样来增强视觉转换器。结果，令牌的数量不再是恒定的，而是针对每个输入图像而变化。通过将ATS集成为当前Transformer块中的附加层，我们可以将它们转换为具有自适应令牌数量的更高效的视觉转换器。由于ATS是一个无参数模块，因此可以将其作为即插即用模块添加到现成的预训练视觉转换器中，从而在不进行任何额外培训的情况下降低GFLOP。此外，由于其可区分的设计，还可以训练配备有ATS的视觉Transformer。我们通过将我们的模块添加到多个SOTA视觉转换器来评估图像和视频分类任务。我们提出的模块通过将计算成本（GFLOPs）降低2倍来改进SOTA，同时保留ImageNet，Kinetics-400和Kinetics-600数据集上SOTA模型的准确性。
. While state-of-the-art vision transformer models achieve promising results for image classification, they are computationally expensive and require many GFLOPs. Although the GFLOPs of a vision trans-former can be decreased by reducing the number of tokens in the network, there is no setting that is optimal for all input images. In this work, we therefore introduce a differentiable parameter-free Adaptive Token Sampling (ATS) module, which can be plugged into any existing vision transformer architecture. ATS empowers vision transformers by scoring and adaptively sampling significant tokens. As a result, the number of tokens is not constant anymore and varies for each input image. By integrating ATS as an additional layer within current transformer blocks, we can convert them into much more efficient vision transformers with an adaptive number of tokens. Since ATS is a parameter-free module, it can be added to off-the-shelf pre-trained vision transformers as a plug and play module, thus reducing their GFLOPs without any additional training. Moreover, due to its differentiable design, one can also train a vision transformer equipped with ATS. We evaluate our module on both image and video classification tasks by adding it to multiple SOTA vision transformers. Our proposed module improves the SOTA by reducing the computational cost (GFLOPs) by 2 × , while preserving the accuracy of SOTA models on ImageNet, Kinetics-400 and Kinetics-600 datasets.