Dict-BERT: Enhancing Language Model Pre-training with Dictionary
Dict-BERT: Enhancing Language Model Pre-training with Dictionary
复制标题
DOI：
10.18653/v1/2022.findings-acl.150
复制
发表时间：
2021-10
期刊：
影响因子：
--
通讯作者：
W. Yu;Chenguang Zhu;Yuwei Fang;Donghan Yu;Shuohang Wang;Yichong Xu;Michael Zeng;Meng Jiang
中科院分区：
文献类型：
--
作者：
W. Yu;Chenguang Zhu;Yuwei Fang;Donghan Yu;Shuohang Wang;Yichong Xu;Michael Zeng;Meng Jiang
预训练的语言模型（PLM）旨在通过对大规模语料库进行自学培训任务来学习通用语言表示。由于PLM在不同上下文中捕获单词语义，因此单词表示的质量在很大程度上取决于单词频率，这通常遵循预训练语料库中的重尾分布。因此，尾巴上稀有单词的嵌入通常优化不佳。在这项工作中，我们专注于通过利用字典中稀有词的定义（例如Wiktionary）来增强语言模型的预训练。为了将罕见的单词定义作为输入的一部分，我们从字典中获取其定义，并将其附加到输入文本序列的末尾。除了使用蒙版语言建模目标培训外，我们还提出了两个新颖的自我监管的预训练的预训练任务，并在输入文本序列和稀有单词定义之间的句子和句子级对齐中，以增强用字典增强语言建模表示。我们评估了有关语言理解基准胶和八个专业域基准数据集的拟议的dict-bert模型。广泛的实验表明，dict-bert可以显着提高对稀有单词的理解，并在各种NLP下游任务上提高模型性能。
Pre-trained language models (PLMs) aim to learn universal language representations by conducting self-supervised training tasks on large-scale corpora. Since PLMs capture word semantics in different contexts, the quality of word representations highly depends on word frequency, which usually follows a heavy-tailed distributions in the pre-training corpus. Therefore, the embeddings of rare words on the tail are usually poorly optimized. In this work, we focus on enhancing language model pre-training by leveraging definitions of the rare words in dictionaries (e.g., Wiktionary). To incorporate a rare word definition as a part of input, we fetch its definition from the dictionary and append it to the end of the input text sequence. In addition to training with the masked language modeling objective, we propose two novel self-supervised pre-training tasks on word and sentence-level alignment between input text sequence and rare word definitions to enhance language modeling representation with dictionary. We evaluate the proposed Dict-BERT model on the language understanding benchmark GLUE and eight specialized domain benchmark datasets. Extensive experiments demonstrate that Dict-BERT can significantly improve the understanding of rare words and boost model performance on various NLP downstream tasks.