How LSTM Encodes Syntax: Exploring Context Vectors and Semi-Quantization on Natural Text
How LSTM Encodes Syntax: Exploring Context Vectors and Semi-Quantization on Natural Text
复制标题
DOI：
10.18653/v1/2020.coling-main.356
复制
发表时间：
2020-10
期刊：
影响因子：
--
通讯作者：
Chihiro Shibata;Kei Uchiumi;D. Mochihashi
中科院分区：
文献类型：
--
作者：
Chihiro Shibata;Kei Uchiumi;D. Mochihashi
长短期记忆递归神经网络（LSTM）被广泛使用，并且已知用于捕获信息性的长期语法依赖关系。然而，这些信息是如何反映在其内部向量的自然文本尚未得到充分的研究。我们通过学习一种语言模型来分析它们，在这种语言模型中，语法结构是隐式给出的。我们根据经验表明，上下文更新向量，即内部门的输出，近似量化为二进制或三进制值，以帮助语言模型准确地计算嵌套的深度，正如Suzgun等人（2019）最近为合成Dyck语言所展示的那样。对于上下文向量中的某些维度，我们发现它们的激活与短语结构的深度高度相关，如VP和NP。此外，通过L1正则化，我们还发现它可以从上下文向量的少量分量准确地预测单词是否在短语结构中。即使是从原始文本中学习的情况下，上下文向量仍然与短语结构相关。最后，我们证明了功能词和触发短语的部分语音的自然集群表示在LSTM的上下文更新向量的一个小但主要的子空间中。
Long Short-Term Memory recurrent neural network (LSTM) is widely used and known to capture informative long-term syntactic dependencies. However, how such information are reflected in its internal vectors for natural text has not yet been sufficiently investigated. We analyze them by learning a language model where syntactic structures are implicitly given. We empirically show that the context update vectors, i.e. outputs of internal gates, are approximately quantized to binary or ternary values to help the language model to count the depth of nesting accurately, as Suzgun et al. (2019) recently show for synthetic Dyck languages. For some dimensions in the context vector, we show that their activations are highly correlated with the depth of phrase structures, such as VP and NP. Moreover, with an L1 regularization, we also found that it can accurately predict whether a word is inside a phrase structure or not from a small number of components of the context vector. Even for the case of learning from raw text, context vectors are shown to still correlate well with the phrase structures. Finally, we show that natural clusters of the functional words and the part of speeches that trigger phrases are represented in a small but principal subspace of the context-update vector of LSTM.