PaCa-ViT: Learning Patch-to-Cluster Attention in Vision Transformers
PaCa-ViT: Learning Patch-to-Cluster Attention in Vision Transformers
复制标题
DOI：
10.1109/cvpr52729.2023.01781
复制
发表时间：
2022-03
期刊：
影响因子：
--
通讯作者：
Ryan Grainger;Thomas Paniagua;Xi Song;Naresh P. Cuntoor;Mun Wai Lee;Tianfu Wu
中科院分区：
文献类型：
--
作者：
Ryan Grainger;Thomas Paniagua;Xi Song;Naresh P. Cuntoor;Mun Wai Lee;Tianfu Wu
视觉变形器(VITS)建立在将图像补丁视为“视觉标记”的假设之上，并学习补丁到补丁的注意。基于补丁嵌入的标记器相对于其对应的文本标记器具有语义鸿沟。补丁到补丁的注意受到二次复杂性问题的困扰，也使得解释学习VITS变得不是平凡的。为了解决VIT中的这些问题，本文提出在VIT中学习补丁到簇注意(PACA)。我们的Paca-vit中的查询从补丁开始，而键和值直接基于集群(具有预定义的少量集群)。集群是端到端学习的，从而产生更好的标记器，并诱导联合集群以获得注意力和注意力集群，以获得更好和可解释的模型。将二次复杂度放宽为线性复杂度。提出的PACA模块用于设计高效和可解释的VIT主干和语义分割头网络。在实验中，对ImageNet-1k图像分类、MS-CoCo目标检测和实例分割以及MIT-ADE20k语义分割进行了测试。与现有技术相比，它在所有三个基准测试中都获得了比Swin[32]和PVTS[47]、[48]更好的性能，在ImageNet-1k和MIT-ADE20k中有明显的差距。由于线性复杂性，它在MS-COCO和MIT-ADE20k上的效率也明显高于PVT模型。学习的簇在语义上是有意义的。代码和模型检查点可在https：/githorb.com/iVMCL/PaCaViT上找到。
Vision Transformers (ViTs) are built on the assumption of treating image patches as “visual tokens” and learn patch-to-patch attention. The patch embedding based tokenizer has a semantic gap with respect to its counterpart, the textual tokenizer. The patch-to-patch attention suffers from the quadratic complexity issue, and also makes it non-trivial to explain learned ViTs. To address these issues in ViT, this paper proposes to learn Patch-to-Cluster attention (PaCa) in ViT. Queries in our PaCa-ViT starts with patches, while keys and values are directly based on clustering (with a predefined small number of clusters). The clusters are learned end-to-end, leading to better tokenizers and inducing joint clustering-for-attention and attention-for-clustering for better and interpretable models. The quadratic complexity is relaxed to linear complexity. The proposed PaCa module is used in designing efficient and interpretable ViT backbones and semantic segmentation head networks. In experiments, the proposed methods are tested on ImageNet-1k image classification, MS-COCO object detection and instance segmentation and MIT-ADE20k semantic segmentation. Compared with the prior art, it obtains better performance in all the three benchmarks than the SWin [32] and the PVTs [47], [48] by significant margins in ImageNet-1k and MIT-ADE20k. It is also significantly more efficient than PVT models in MS-COCO and MIT-ADE20k due to the linear complexity. The learned clusters are semantically meaningful. Code and model checkpoints are available at https:/github.com/iVMCL/PaCaViT.