An Empirical Study of Pre-trained Transformers for Arabic Information Extraction
An Empirical Study of Pre-trained Transformers for Arabic Information Extraction
复制标题
DOI：
10.18653/v1/2020.emnlp-main.382
复制
发表时间：
2020-04
期刊：
影响因子：
--
通讯作者：
Wuwei Lan;Yang Chen;Wei Xu;Alan Ritter
中科院分区：
文献类型：
--
作者：
Wuwei Lan;Yang Chen;Wei Xu;Alan Ritter
多语言预训练的变形器，如mBERT （Devlin等人，2019）和XLM-RoBERTa (Conneau等人，2020a)，已被证明可以实现有效的跨语言零shot迁移。然而，它们在阿拉伯文信息提取（IE）任务中的表现并没有得到很好的研究。在本文中，我们预训练了一个定制的双语BERT，称为GigaBERT，专门为阿拉伯语NLP和英语到阿拉伯语零射击迁移学习而设计。我们在四个IE任务中研究了GigaBERT在零短传递方面的有效性：命名实体识别、词性标记、参数角色标记和关系提取。我们的最佳模型在监督和零射击转移设置中都显著优于mBERT， XLM-RoBERTa和AraBERT （Antoun等人，2020）。我们已经在https://github.com/lanwuwei/GigaBERT上公开了我们的预训练模型。
Multilingual pre-trained Transformers, such as mBERT (Devlin et al., 2019) and XLM-RoBERTa (Conneau et al., 2020a), have been shown to enable the effective cross-lingual zero-shot transfer. However, their performance on Arabic information extraction (IE) tasks is not very well studied. In this paper, we pre-train a customized bilingual BERT, dubbed GigaBERT, that is designed specifically for Arabic NLP and English-to-Arabic zero-shot transfer learning. We study GigaBERT's effectiveness on zero-short transfer across four IE tasks: named entity recognition, part-of-speech tagging, argument role labeling, and relation extraction. Our best model significantly outperforms mBERT, XLM-RoBERTa, and AraBERT (Antoun et al., 2020) in both the supervised and zero-shot transfer settings. We have made our pre-trained models publicly available at https://github.com/lanwuwei/GigaBERT.