Bidirectional Transformer with absolute-position aware relative position encoding for encoding sentences
Bidirectional Transformer with absolute-position aware relative position encoding for encoding sentences
复制标题
DOI：
10.1007/s11704-022-0610-2
复制
发表时间：
2022-08
影响因子：
4.2
通讯作者：
Le Qi;Yu Zhang;Ting Liu
中科院分区：
文献类型：
--
作者：
Le Qi;Yu Zhang;Ting Liu
变形金刚在许多自然语言处理任务中得到了广泛的研究，它利用多头注意和位置前馈网络，可以从整个句子中捕获依赖关系，并且具有很高的并行性。然而，变压器的上述两个组成部分是位置无关的，这就导致变压器在句子结构建模方面的能力较弱。现有研究通常采用位置编码或掩码策略来获取句子的结构信息。在本文中，我们旨在从三个方面加强变形器对句子线性结构的建模能力，包括符号的绝对位置、相对距离和符号之间的方向。我们提出了一种结合位置编码和掩码策略的具有绝对位置感知的相对位置编码的双向变压器（BiAR-Transformer）。我们通过一种新颖的绝对位置感知相对位置编码来建模标记之间的相对距离以及标记的绝对位置。同时，我们采用双向掩码策略对令牌之间的方向进行建模。在自然语言推理、意译识别、情感分类和机器翻译任务上的实验结果表明，BiAR-Transformer的性能优于其他强基线。
Transformers have been widely studied in many natural language processing (NLP) tasks, which can capture the dependency from the whole sentence with a high parallelizability thanks to the multi-head attention and the position-wise feed-forward network. However, the above two components of transformers are position-independent, which causes transformers to be weak in modeling sentence structures. Existing studies commonly utilized positional encoding or mask strategies for capturing the structural information of sentences. In this paper, we aim at strengthening the ability of transformers on modeling the linear structure of sentences from three aspects, containing the absolute position of tokens, the relative distance, and the direction between tokens. We propose a novel bidirectional Transformer with absolute-position aware relative position encoding (BiAR-Transformer) that combines the positional encoding and the mask strategy together. We model the relative distance between tokens along with the absolute position of tokens by a novel absolute-position aware relative position encoding. Meanwhile, we apply a bidirectional mask strategy for modeling the direction between tokens. Experimental results on the natural language inference, paraphrase identification, sentiment classification and machine translation tasks show that BiAR-Transformer achieves superior performance than other strong baselines.