A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space
A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space
复制标题
DOI：
10.18653/v1/2021.emnlp-main.471
复制
发表时间：
2021-09
期刊：
影响因子：
--
通讯作者：
Alex Jones;W. Wang;Kyle Mahowald
中科院分区：
文献类型：
--
作者：
Alex Jones;W. Wang;Kyle Mahowald
在跨语性语言模型中，许多不同语言的表示形式生活在同一空间中。在这里，我们研究了影响句子级别的语言和非语言因素，这些因素影响了101种语言和5,050种语言对的跨语言审慎语言模型。我们以基于BERT的LABSE和基于Bilstm的激光为我们的模型，而圣经作为我们的语料库，以Bitext检索性能的形式计算基于任务的跨语性对齐方式，以及矢量空间对准和同构的四个固有度量。然后，我们研究了一系列语言，准语言和与训练相关的特征，作为这些对齐指标的潜在预测指标。我们的分析结果表明，形态复杂性中的单词顺序一致和一致性是跨语言最强的语言预测指标之一。我们还指出，在家庭培训数据中比全面的语言特定培训数据更强大。除了研究单词顺序一致对同构对同构的影响外，我们还通过研究形态分割对英语inuktitut对准的影响来验证我们的一些语言发现。我们为我们的实验提供数据和代码。
In cross-lingual language models, representations for many different languages live in the same space. Here, we investigate the linguistic and non-linguistic factors affecting sentence-level alignment in cross-lingual pretrained language models for 101 languages and 5,050 language pairs. Using BERT-based LaBSE and BiLSTM-based LASER as our models, and the Bible as our corpus, we compute a task-based measure of cross-lingual alignment in the form of bitext retrieval performance, as well as four intrinsic measures of vector space alignment and isomorphism. We then examine a range of linguistic, quasi-linguistic, and training-related features as potential predictors of these alignment metrics. The results of our analyses show that word order agreement and agreement in morphological complexity are two of the strongest linguistic predictors of cross-linguality. We also note in-family training data as a stronger predictor than language-specific training data across the board. We verify some of our linguistic findings by looking at the effect of morphological segmentation on English-Inuktitut alignment, in addition to examining the effect of word order agreement on isomorphism for 66 zero-shot language pairs from a different corpus. We make the data and code for our experiments publicly available.