Assessing Phrasal Representation and Composition in Transformers
Assessing Phrasal Representation and Composition in Transformers
复制标题
DOI：
10.18653/v1/2020.emnlp-main.397
复制
发表时间：
2020-10
期刊：
影响因子：
--
通讯作者：
Lang-Chi Yu;Allyson Ettinger
中科院分区：
文献类型：
--
作者：
Lang-Chi Yu;Allyson Ettinger
深度 Transformer 模型将 NLP 任务的性能推向了新的极限，建议对复杂的语言输入（例如短语）进行复杂的处理。然而，我们对这些模型如何处理短语表示以及这是否反映了像人类那样复杂的短语含义组成的理解有限。在本文中，我们对最先进的预训练变压器中的短语表示进行了系统分析。我们使用利用人类对短语相似性和意义转移的判断进行测试，并比较单词重叠控制前后的结果，以区分词汇效果与构图效果。我们发现这些模型中的短语表示很大程度上依赖于单词内容，几乎没有细微差别的证据。我们还识别了跨模型、层和表示类型的短语表示质量的变化，并就这些模型的表示的使用提出了相应的建议。
Deep transformer models have pushed performance on NLP tasks to new limits, suggesting sophisticated treatment of complex linguistic inputs, such as phrases. However, we have limited understanding of how these models handle representation of phrases, and whether this reflects sophisticated composition of phrase meaning like that done by humans. In this paper, we present systematic analysis of phrasal representations in state-of-the-art pre-trained transformers. We use tests leveraging human judgments of phrase similarity and meaning shift, and compare results before and after control of word overlap, to tease apart lexical effects versus composition effects. We find that phrase representation in these models relies heavily on word content, with little evidence of nuanced composition. We also identify variations in phrase representation quality across models, layers, and representation types, and make corresponding recommendations for usage of representations from these models.