Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in Number and Gender Assignment
Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in Number and Gender Assignment
复制标题
分析神经语言模型：上下文分解揭示了数字和性别分配中的默认推理
DOI：
10.18653/v1/k19-1001
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
Dieuwke Hupkes
中科院分区：
文献类型：
--
作者：
Jaap Jumelet;Willem H. Zuidema;Dieuwke Hupkes
近期大量研究表明，循环神经网络语言模型能够处理多种语法现象。然而，这些模型如何能够如此出色地完成这些非凡的任务，仍然是一个悬而未决的问题。为了更深入地了解长短期记忆网络（LSTMs）是基于什么信息做出决策的，我们提出了一种语境分解的推广方法（GCD）。特别是，这种设置使我们能够准确地提炼出预测的哪一部分源于语义启发，哪一部分真正来自句法线索，以及哪一部分是由模型自身的偏差所导致的。我们在与句法一致和共指消解相关的任务上研究了这种技术，并发现该模型在执行这些任务时强烈依赖于一种默认推理效应。
Extensive research has recently shown that recurrent neural language models are able to process a wide range of grammatical phenomena. How these models are able to perform these remarkable feats so well, however, is still an open question. To gain more insight into what information LSTMs base their decisions on, we propose a generalisation of Contextual Decomposition (GCD). In particular, this setup enables us to accurately distil which part of a prediction stems from semantic heuristics, which part truly emanates from syntactic cues and which part arise from the model biases themselves instead. We investigate this technique on tasks pertaining to syntactic agreement and co-reference resolution and discover that the model strongly relies on a default reasoning effect to perform these tasks.
DOI：
10.18653/v1/n18-2003
发表时间：
2018-04
期刊：
ArXiv
影响因子：
--
作者：
Jieyu Zhao;Tianlu Wang;Mark Yatskar;Vicente Ordonez;Kai-Wei Chang
通讯作者：
Jieyu Zhao;Tianlu Wang;Mark Yatskar;Vicente Ordonez;Kai-Wei Chang