Transformer Dissection: An Unified Understanding for Transformer’s Attention via the Lens of Kernel
Transformer Dissection: An Unified Understanding for Transformer’s Attention via the Lens of Kernel
复制标题
DOI：
10.18653/v1/d19-1443
复制
发表时间：
2019-08
期刊：
影响因子：
--
通讯作者：
Yao-Hung Hubert Tsai;Shaojie Bai;M. Yamada;Louis-Philippe Morency;R. Salakhutdinov
中科院分区：
文献类型：
--
作者：
Yao-Hung Hubert Tsai;Shaojie Bai;M. Yamada;Louis-Philippe Morency;R. Salakhutdinov
Transformer是一个功能强大的架构，可以在各种序列学习任务上实现上级性能，包括神经机器翻译、语言理解和序列预测。Transformer的核心是注意力机制，它并发地处理流中的所有输入。在本文中，我们提出了一个新的公式的注意通过透镜的内核。更准确地说，我们意识到注意力可以被看作是对输入应用内核平滑器，内核得分是输入之间的相似性。这个新的公式为我们提供了一个更好的方法来理解Transformer注意力的各个组成部分，例如整合位置嵌入的更好方法。我们基于内核的公式化的另一个重要优点是，它为Transformer的注意力组成的更大空间铺平了道路。作为一个例子，我们提出了一个新的变形Transformer的注意力模型的输入作为一个产品的对称内核。这种方法实现了竞争力的性能，以更少的计算量的最先进的模型的当前状态。在我们的实验中，我们经验性地研究了两个广泛使用的任务：神经机器翻译和序列预测的不同内核构建策略。
Transformer is a powerful architecture that achieves superior performance on various sequence learning tasks, including neural machine translation, language understanding, and sequence prediction. At the core of the Transformer is the attention mechanism, which concurrently processes all inputs in the streams. In this paper, we present a new formulation of attention via the lens of the kernel. To be more precise, we realize that the attention can be seen as applying kernel smoother over the inputs with the kernel scores being the similarities between inputs. This new formulation gives us a better way to understand individual components of the Transformer’s attention, such as the better way to integrate the positional embedding. Another important advantage of our kernel-based formulation is that it paves the way to a larger space of composing Transformer’s attention. As an example, we propose a new variant of Transformer’s attention which models the input as a product of symmetric kernels. This approach achieves competitive performance to the current state of the art model with less computation. In our experiments, we empirically study different kernel construction strategies on two widely used tasks: neural machine translation and sequence prediction.