Investigating Representations of Verb Bias in Neural Language Models
Investigating Representations of Verb Bias in Neural Language Models
复制标题
DOI：
10.18653/v1/2020.emnlp-main.376
复制
发表时间：
2020-10
期刊：
影响因子：
--
通讯作者：
Robert D. Hawkins;Takateru Yamakoshi;T. Griffiths;A. Goldberg
中科院分区：
文献类型：
--
作者：
Robert D. Hawkins;Takateru Yamakoshi;T. Griffiths;A. Goldberg
语言通常提供不止一个语法结构来表达某些类型的消息。众所周知，说话人对结构的选择取决于多种因素，包括主要动词的选择--这一现象被称为\emph{动词偏见}。在这里，我们介绍了DAIS，一个大型基准数据集，包含对英语与格交替中5K个不同句子对的50K个人类判断。该数据集包括200个唯一动词，并系统地改变参数的确定性和长度。我们使用这个数据集，以及一个现有的自然发生的数据语料库，来评估最近的神经语言模型捕获人类偏好的情况。结果表明，较大的模型比较小的模型性能更好，即使在类似的参数和训练设置下，变压器架构(例如GPT-2)的性能也往往优于递归架构(例如LSTM)。对内部特征表征的额外分析表明，转换器可以更好地将特定的词汇信息与语法结构结合起来。
Languages typically provide more than one grammatical construction to express certain types of messages. A speaker's choice of construction is known to depend on multiple factors, including the choice of main verb -- a phenomenon known as \emph{verb bias}. Here we introduce DAIS, a large benchmark dataset containing 50K human judgments for 5K distinct sentence pairs in the English dative alternation. This dataset includes 200 unique verbs and systematically varies the definiteness and length of arguments. We use this dataset, as well as an existing corpus of naturally occurring data, to evaluate how well recent neural language models capture human preferences. Results show that larger models perform better than smaller models, and transformer architectures (e.g. GPT-2) tend to out-perform recurrent architectures (e.g. LSTMs) even under comparable parameter and training settings. Additional analyses of internal feature representations suggest that transformers may better integrate specific lexical information with grammatical constructions.