Frequency Effects on Syntactic Rule Learning in Transformers
Frequency Effects on Syntactic Rule Learning in Transformers
复制标题
Transformer 中句法规则学习的频率效应
DOI：
10.18653/v1/2021.emnlp-main.72
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Ellie Pavlick
中科院分区：
文献类型：
--
作者：
Jason Wei;Dan Garrette;Tal Linzen;Ellie Pavlick
预训练的语言模型在需要象征性推理的各种语言任务上表现出色，这提出了这样的问题，即这种模型是否暗中代表抽象的符号和规则，我们使用伯特对英语主题 - verb协议的绩效进行了研究。与先前的工作不同，我们从头开始训练Bert的多个实例，使我们可以在培训时间进行一系列受控干预措施。通常，在训练中从未发生过的主体 - 动力对，这表明我们还发现，绩效受到了词频率的严重影响，实验表明动词形式的绝对频率是不幸的是，伯特在推理时对这些频率效应进行了更紧密的分析，还揭示了伯特的频率与替代影响相对。行为与一般适用SVA规则的系统一致，但努力克服强大的训练先验并估算不频繁的词汇项目的一致性特征（单数与复数）。
Pre-trained language models perform well on a variety of linguistic tasks that require symbolic reasoning, raising the question of whether such models implicitly represent abstract symbols and rules. We investigate this question using the case study of BERT’s performance on English subject–verb agreement. Unlike prior work, we train multiple instances of BERT from scratch, allowing us to perform a series of controlled interventions at pre-training time. We show that BERT often generalizes well to subject–verb pairs that never occurred in training, suggesting a degree of rule-governed behavior. We also find, however, that performance is heavily influenced by word frequency, with experiments showing that both the absolute frequency of a verb form, as well as the frequency relative to the alternate inflection, are causally implicated in the predictions BERT makes at inference time. Closer analysis of these frequency effects reveals that BERT’s behavior is consistent with a system that correctly applies the SVA rule in general but struggles to overcome strong training priors and to estimate agreement features (singular vs. plural) on infrequent lexical items.
DOI：
10.1146/annurev-linguistics-032020-051035
发表时间：
2021-01-01
期刊：
ANNUAL REVIEW OF LINGUISTICS, VOL 7
影响因子：
--
作者：
Linzen, Tal;Baroni, Marco
通讯作者：
Baroni, Marco