Fine-Tuning BERT with Character-Level Noise for Zero-Shot Transfer to Dialects and Closely-Related Languages
Fine-Tuning BERT with Character-Level Noise for Zero-Shot Transfer to Dialects and Closely-Related Languages
复制标题
DOI：
10.48550/arxiv.2303.17683
复制
发表时间：
2023-03
期刊：
影响因子：
--
通讯作者：
Aarohi Srivastava;David Chiang-
中科院分区：
文献类型：
--
作者：
Aarohi Srivastava;David Chiang-
在这项工作中，我们在微调BERT时以各种形式诱导字符级噪声，以实现零拍摄跨语言迁移到看不见的方言和语言。我们在三个高级分类任务上微调BERT，并在各种看不见的方言和语言上评估我们的方法。我们发现，字符级噪声可以是一个非常有效的代理跨语言迁移在某些条件下，而在其他人没有帮助。具体来说，我们在任务的性质以及源语言和目标语言之间的关系方面探索了这些差异，发现在微调期间引入字符级噪声特别有帮助，当任务利用表面级线索并且源-目标跨语言对具有相对较高的词汇重叠时，不太有意义的）平均看不见的标记。
In this work, we induce character-level noise in various forms when fine-tuning BERT to enable zero-shot cross-lingual transfer to unseen dialects and languages. We fine-tune BERT on three sentence-level classification tasks and evaluate our approach on an assortment of unseen dialects and languages. We find that character-level noise can be an extremely effective agent of cross-lingual transfer under certain conditions, while it is not as helpful in others. Specifically, we explore these differences in terms of the nature of the task and the relationships between source and target languages, finding that introduction of character-level noise during fine-tuning is particularly helpful when a task draws on surface level cues and the source-target cross-lingual pair has a relatively high lexical overlap with shorter (i.e., less meaningful) unseen tokens on average.