Grammar Induction with Neural Language Models: An Unusual Replication
Grammar Induction with Neural Language Models: An Unusual Replication
复制标题
神经语言模型的语法归纳：不寻常的复制
DOI：
--
复制
发表时间：
2018
期刊：
影响因子：
--
通讯作者：
Samuel R. Bowman
中科院分区：
文献类型：
--
作者：
Phu Mon Htut;Kyunghyun Cho;Samuel R. Bowman
最近关于潜在树学习的大量工作试图开发具有解析值潜在变量的神经网络模型，并在非解析任务上训练它们，希望让它们发现可解释的树结构。在最近的一篇论文中，Shen等人（2018）介绍了这样一个模型，并在语言建模的目标任务上报告了接近最先进的结果，以及在选区解析上的第一个强潜在树学习结果。在试图重现这些结果的过程中，我们发现了使原始结果难以信任的问题，包括对有效测试集的调优甚至训练。在这里，我们试图在一个公平的实验中重现这些结果，并将它们扩展到两个新的数据集。我们发现这项工作的结果是鲁棒的：所研究的模型的所有变体都优于所有潜在树学习基线，并且与符号语法归纳系统竞争。我们发现该模型代表了潜在树学习的第一个经验成功，并且神经网络语言建模作为语法归纳的设置值得进一步研究。
A substantial thread of recent work on latent tree learning has attempted to develop neural network models with parse-valued latent variables and train them on non-parsing tasks, in the hope of having them discover interpretable tree structure. In a recent paper, Shen et al. (2018) introduce such a model and report near-state-of-the-art results on the target task of language modeling, and the first strong latent tree learning result on constituency parsing. In an attempt to reproduce these results, we discover issues that make the original results hard to trust, including tuning and even training on what is effectively the test set. Here, we attempt to reproduce these results in a fair experiment and to extend them to two new datasets. We find that the results of this work are robust: All variants of the model under study outperform all latent tree learning baselines, and perform competitively with symbolic grammar induction systems. We find that this model represents the first empirical success for latent tree learning, and that neural network language modeling warrants further study as a setting for grammar induction.