Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformers
Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformers
复制标题
DOI：
10.48550/arxiv.2205.08078
复制
发表时间：
2022-05
期刊：
影响因子：
--
通讯作者：
Arda Sahiner;Tolga Ergen;Batu Mehmet Ozturkler;J. Pauly;M. Mardani;Mert Pilanci
中科院分区：
文献类型：
--
作者：
Arda Sahiner;Tolga Ergen;Batu Mehmet Ozturkler;J. Pauly;M. Mardani;Mert Pilanci
使用自我注意或其提出的替代方案的视觉变换器在许多图像相关的任务中已经表现出有希望的结果。然而，注意力的基础归纳偏差并没有得到很好的理解。针对这一问题，本文从凸对偶性的透镜来分析注意。对于非线性点积自注意，和替代机制，如MLP混合器和傅立叶神经算子（FNO），我们推导出等价的有限维凸问题，是可解释的，可解的全局最优性。凸规划导致{\it block nuclear-norm regularization}，这促进了潜在特征和令牌维度的低秩。特别是，我们展示了自我注意力网络如何根据它们的潜在相似性隐式地对令牌进行聚类。我们进行实验，通过微调各种凸注意力头来转移预先训练的Transformer骨干用于CIFAR-100分类。结果表明，与现有的MLP或线性头相比，由注意引起的偏见的优点。
Vision transformers using self-attention or its proposed alternatives have demonstrated promising results in many image related tasks. However, the underpinning inductive bias of attention is not well understood. To address this issue, this paper analyzes attention through the lens of convex duality. For the non-linear dot-product self-attention, and alternative mechanisms such as MLP-mixer and Fourier Neural Operator (FNO), we derive equivalent finite-dimensional convex problems that are interpretable and solvable to global optimality. The convex programs lead to {\it block nuclear-norm regularization} that promotes low rank in the latent feature and token dimensions. In particular, we show how self-attention networks implicitly clusters the tokens, based on their latent similarity. We conduct experiments for transferring a pre-trained transformer backbone for CIFAR-100 classification by fine-tuning a variety of convex attention heads. The results indicate the merits of the bias induced by attention compared with the existing MLP or linear heads.