Structural Supervision Improves Learning of Non-Local Grammatical Dependencies
Structural Supervision Improves Learning of Non-Local Grammatical Dependencies
复制标题
结构监督改善非局部语法依赖的学习
DOI：
10.18653/v1/n19-1334
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
R. Levy
中科院分区：
文献类型：
--
作者：
Ethan Gotlieb Wilcox;Peng Qian;Richard Futrell;Miguel Ballesteros;R. Levy
在大型语料库上训练的最先进的LSTM语言模型以令人印象深刻的细节学习顺序偶然性，并已被证明获得了一些非局部语法依赖性并取得了一些成功。在这里，我们调查是否监督与层次结构增强学习的一系列语法依赖关系，以前只针对主谓一致的问题。使用心理语言学的对照实验方法，我们比较了基于单词的LSTM模型与递归神经网络语法（RNNG）的性能。（Dyer et al. 2016），它代表了层次句法结构，并使用神经控制将其部署在从左到右的处理中，在英语中的两类非本地语法依赖-负极性许可和填充-间隙依赖-测试在一系列配置。对于两个模型使用相同的训练数据，我们发现RNNG在两种类型的语法依赖关系上都优于LSTM，甚至学习了许多关于填充缺口依赖关系的岛屿约束。因此，结构监督在获取有关非局部语法依赖关系的类似人类的概括方面，比纯粹基于字符串的神经语言模型训练提供了数据效率优势。
State-of-the-art LSTM language models trained on large corpora learn sequential contingencies in impressive detail, and have been shown to acquire a number of non-local grammatical dependencies with some success. Here we investigate whether supervision with hierarchical structure enhances learning of a range of grammatical dependencies, a question that has previously been addressed only for subject-verb agreement. Using controlled experimental methods from psycholinguistics, we compare the performance of word-based LSTM models versus Recurrent Neural Network Grammars (RNNGs) (Dyer et al. 2016) which represent hierarchical syntactic structure and use neural control to deploy it in left-to-right processing, on two classes of non-local grammatical dependencies in English—Negative Polarity licensing and Filler-Gap Dependencies—tested in a range of configurations. Using the same training data for both models, we find that the RNNG outperforms the LSTM on both types of grammatical dependencies and even learns many of the Island Constraints on the filler-gap dependency. Structural supervision thus provides data efficiency advantages over purely string-based training of neural language models in acquiring human-like generalizations about non-local grammatical dependencies.
DOI：
10.1037/h0087426
发表时间：
2003-09-01
影响因子：
1.3
作者：
Masson, MEJ;Loftus, GR
通讯作者：
Loftus, GR