Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERT
Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERT
复制标题
DOI：
10.18653/v1/2021.eacl-main.215
复制
发表时间：
2021-01
期刊：
影响因子：
--
通讯作者：
Isabel Papadimitriou;Ethan A. Chi;Richard Futrell;Kyle Mahowald
中科院分区：
文献类型：
--
作者：
Isabel Papadimitriou;Ethan A. Chi;Richard Futrell;Kyle Mahowald
我们研究了多种语言Bert（Mbert）如何通过研究形态同步对齐的高阶语法特征（不同的语言如何定义了“主题”）的高阶语法特征来编码语法以及形态同步一致性如何影响上下文嵌入空间，我们训练分类器以恢复Mbert嵌入的主题（这些主题）不要包含有关形态同步的明显信息），然后在不同步的句子（主题分类取决于对齐方式）上评估它们，我们发现所得的分布式分布反映了他们的训练语言结果表明，姆伯特表示受到任何一个输入句子中未表现出的高级语法特征的影响，这就是跨语言的强大研究。关于语义和话语因素，正如许多功能语言学文献所提出的那样。深入了解语法特征如何在上下文嵌入空间中表现出来，这是先前工作未涵盖的抽象水平。
We investigate how Multilingual BERT (mBERT) encodes grammar by examining how the high-order grammatical feature of morphosyntactic alignment (how different languages define what counts as a “subject”) is manifested across the embedding spaces of different languages. To understand if and how morphosyntactic alignment affects contextual embedding spaces, we train classifiers to recover the subjecthood of mBERT embeddings in transitive sentences (which do not contain overt information about morphosyntactic alignment) and then evaluate them zero-shot on intransitive sentences (where subjecthood classification depends on alignment), within and across languages. We find that the resulting classifier distributions reflect the morphosyntactic alignment of their training languages. Our results demonstrate that mBERT representations are influenced by high-level grammatical features that are not manifested in any one input sentence, and that this is robust across languages. Further examining the characteristics that our classifiers rely on, we find that features such as passive voice, animacy and case strongly correlate with classification decisions, suggesting that mBERT does not encode subjecthood purely syntactically, but that subjecthood embedding is continuous and dependent on semantic and discourse factors, as is proposed in much of the functional linguistics literature. Together, these results provide insight into how grammatical features manifest in contextual embedding spaces, at a level of abstraction not covered by previous work.