A Systematic Assessment of Syntactic Generalization in Neural Language Models
A Systematic Assessment of Syntactic Generalization in Neural Language Models
复制标题
神经语言模型中句法泛化的系统评估
DOI：
10.18653/v1/2020.acl-main.158
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
R. Levy
中科院分区：
文献类型：
--
作者：
Jennifer Hu;Jon Gauthier;Peng Qian;Ethan Gotlieb Wilcox;R. Levy
虽然最先进的神经网络模型在语言建模基准上继续获得较低的困惑分数，但针对广泛覆盖的预测性能进行优化是否会带来类似人类的句法知识仍是未知的。此外，现有的工作还没有提供关于产生适当的句法概括所需的模型属性的清晰图景。我们对神经语言模型的句法知识进行了系统的评估，在一组34个英语句法测试套件上测试了20种模型类型和数据大小的组合。我们发现，不同的模型体系结构在句法泛化性能上存在很大差异，顺序模型的表现逊于其他体系结构。在析因操纵模型体系结构和训练数据集大小(1M-4000万字)的情况下，我们发现对于我们实验中测试的语料库，句法概括性能的变异性因体系结构而异于数据集大小。我们的结果还揭示了困惑与句法概括绩效之间的分离关系。
While state-of-the-art neural network models continue to achieve lower perplexity scores on language modeling benchmarks, it remains unknown whether optimizing for broad-coverage predictive performance leads to human-like syntactic knowledge. Furthermore, existing work has not provided a clear picture about the model properties required to produce proper syntactic generalizations. We present a systematic evaluation of the syntactic knowledge of neural language models, testing 20 combinations of model types and data sizes on a set of 34 English-language syntactic test suites. We find substantial differences in syntactic generalization performance by model architecture, with sequential models underperforming other architectures. Factorially manipulating model architecture and training dataset size (1M-40M words), we find that variability in syntactic generalization performance is substantially greater by architecture than by dataset size for the corpora tested in our experiments. Our results also reveal a dissociation between perplexity and syntactic generalization performance.
DOI：
10.1162/tacl_a_00290
发表时间：
2019-01-01
影响因子：
10.9
作者：
Warstadt, Alex;Singh, Amanpreet;Bowman, Samuel R.
通讯作者：
Bowman, Samuel R.
BLiMP：英语语言最小对的基准
DOI：
10.1162/tacl_a_00321
发表时间：
2020
影响因子：
10.9
作者：
Warstadt, Alex;Parrish, Alicia;Liu, Haokun;Mohananey, Anhad;Peng, Wei;Wang, Sheng-Fu;Bowman, Samuel R.
通讯作者：
Bowman, Samuel R.