Theoretical Limitations of Self-Attention in Neural Sequence Models
Theoretical Limitations of Self-Attention in Neural Sequence Models
复制标题
DOI：
10.1162/tacl_a_00306
复制
发表时间：
2020-01-01
影响因子：
10.9
通讯作者：
Hahn, Michael
中科院分区：
文献类型：
--
作者：
Hahn, Michael
Transformers正在成为NLP的新主力，在各种任务中取得了巨大的成功。与LSTM不同，transformer完全通过自我注意来处理输入序列。以前的工作表明，自我注意处理层次结构的计算能力是有限的。在这项工作中，我们从数学上研究了自注意力模型的计算能力。在软注意和硬注意中，我们展示了自我注意计算能力的强大理论局限性，发现它不能模拟周期性有限状态语言，也不能模拟层次结构，除非层或头的数量随着输入长度的增加而增加。考虑到自我注意的实际成功和语言学中层次结构的突出作用，这些局限性似乎令人惊讶，这表明自然语言可以很好地近似于理论语言学中通常假设的形式语言的模型。
Transformers are emerging as the new workhorse of NLP, showing great success across tasks. Unlike LSTMs, transformers process input sequences entirely through self-attention. Previous work has suggested that the computational capabilities of self-attention to process hierarchical structures are limited. In this work, we mathematically investigate the computational power of self-attention to model formal languages. Across both soft and hard attention, we show strong theoretical limitations of the computational abilities of selfattention, finding that it cannot model periodic finite- state languages, nor hierarchical structure, unless the number of layers or heads increases with input length. These limitations seem surprising given the practical success of self-attention and the prominent role assigned to hierarchical structure in linguistics, suggesting that natural language can be approximated well with models that are too weak for the formal languages typically assumed in theoretical linguistics.