A Targeted Assessment of Incremental Processing in Neural Language Models and Humans
A Targeted Assessment of Incremental Processing in Neural Language Models and Humans
复制标题
神经语言模型和人类增量处理的有针对性的评估
DOI：
--
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
R. Levy
中科院分区：
文献类型：
--
作者：
Ethan Gotlieb Wilcox;P. Vani;R. Levy
我们通过收集16个不同句法测试套件的按词反应时数据，对人类和神经语言模型的增量处理进行了有针对性的放大比较，涉及一系列结构现象。人类反应时的数据来自一种名为内插迷宫任务的新的在线实验范式。我们比较了四种当代语言模型的人类反应时间与逐字概率，这些模型具有不同的体系结构，并在一系列数据集大小上进行了训练。我们发现，在许多现象中，人类和语言模型在非语法句子区域的加工难度都增加了，人类和模型的准确度得分与Marvin和Linzen(2018)大致相当。然而，尽管语言模型的输出在方向上与人类匹配，但我们表明，模型系统地低估了语法句子和非语法句子之间递增处理难度的差异。具体地说，当模型遇到句法违规时，它们无法准确预测在人类数据中观察到的更长的阅读时间。这些结果令人质疑当代语言模型对句法违规的敏感度是否接近人类的表现。
We present a targeted, scaled-up comparison of incremental processing in humans and neural language models by collecting by-word reaction time data for sixteen different syntactic test suites across a range of structural phenomena. Human reaction time data comes from a novel online experimental paradigm called the Interpolated Maze task. We compare human reaction times to by-word probabilities for four contemporary language models, with different architectures and trained on a range of data set sizes. We find that across many phenomena, both humans and language models show increased processing difficulty in ungrammatical sentence regions with human and model ‘accuracy’ scores a la Marvin and Linzen (2018) about equal. However, although language model outputs match humans in direction, we show that models systematically under-predict the difference in magnitude of incremental processing difficulty between grammatical and ungrammatical sentences. Specifically, when models encounter syntactic violations they fail to accurately predict the longer reading times observed in the human data. These results call into question whether contemporary language models are approaching human-like performance for sensitivity to syntactic violations.
BLiMP：英语语言最小对的基准
DOI：
10.1162/tacl_a_00321
发表时间：
2020
影响因子：
10.9
作者：
Warstadt, Alex;Parrish, Alicia;Liu, Haokun;Mohananey, Anhad;Peng, Wei;Wang, Sheng-Fu;Bowman, Samuel R.
通讯作者：
Bowman, Samuel R.