Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations
Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations
复制标题
语言模型中的结构持久性：作为抽象语言表示的窗口
DOI：
10.1162/tacl_a_00504
复制
发表时间：
2021
影响因子：
10.9
通讯作者：
R. Fernández
中科院分区：
文献类型：
--
作者：
Arabella J. Sinclair;Jaap Jumelet;Willem H. Zuidema;R. Fernández
摘要我们研究了现代神经语言模型对结构启动的敏感程度，结构启动是指句子的结构使后续句子中的相同结构更有可能出现的现象。我们探讨了如何启动可以用来研究这些模型的潜力，学习抽象的结构信息，这是一个先决条件，需要自然语言理解技能的任务表现良好。我们引入了一种新的度量和释放Prime-LM，一个大型语料库，我们控制各种语言因素与启动强度。我们发现，Transformer模型确实显示出结构启动的证据，但他们学到的概括在一定程度上受到语义信息的调制。我们的实验还表明，由模型获得的表示可能不仅编码抽象的顺序结构，但涉及一定程度的层次句法信息。更一般地说，我们的研究表明，启动范式是一个有用的，额外的工具，用于深入了解语言模型的能力，并为未来探索模型内部状态的基于启动的调查打开了大门。
Abstract We investigate the extent to which modern neural language models are susceptible to structural priming, the phenomenon whereby the structure of a sentence makes the same structure more probable in a follow-up sentence. We explore how priming can be used to study the potential of these models to learn abstract structural information, which is a prerequisite for good performance on tasks that require natural language understanding skills. We introduce a novel metric and release Prime-LM, a large corpus where we control for various linguistic factors that interact with priming strength. We find that Transformer models indeed show evidence of structural priming, but also that the generalizations they learned are to some extent modulated by semantic information. Our experiments also show that the representations acquired by the models may not only encode abstract sequential structure but involve certain level of hierarchical syntactic information. More generally, our study shows that the priming paradigm is a useful, additional tool for gaining insights into the capacities of language models and opens the door to future priming-based investigations that probe the model’s internal states.1
BLiMP：英语语言最小对的基准
DOI：
10.1162/tacl_a_00321
发表时间：
2020
影响因子：
10.9
作者：
Warstadt, Alex;Parrish, Alicia;Liu, Haokun;Mohananey, Anhad;Peng, Wei;Wang, Sheng-Fu;Bowman, Samuel R.
通讯作者：
Bowman, Samuel R.
调查 BERT 的语言知识：使用 NPI 的五种分析方法
DOI：
10.18653/v1/d19-1286
发表时间：
2019
期刊：
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP
影响因子：
--
作者：
Warstadt, Alex;Cao, Yu;Grosu, Ioana;Peng, Wei;Blix, Hagen;Nie, Yining;Alsop, Anna;Bordia, Shikha;Liu, Haokun;Parrish, Alicia
通讯作者：
Parrish, Alicia