An Empirical Study of the Downstream Reliability of Pre-Trained Word Embeddings
An Empirical Study of the Downstream Reliability of Pre-Trained Word Embeddings
复制标题
DOI：
10.18653/v1/2020.coling-main.299
复制
发表时间：
2020-12
期刊：
影响因子：
--
通讯作者：
Anthony Rios;Brandon Lwowski
中科院分区：
文献类型：
--
作者：
Anthony Rios;Brandon Lwowski
虽然预先训练的单词嵌入已经被证明可以提高下游任务的性能，但其可靠性仍然存在许多问题：同样的预先训练的单词嵌入是否会在训练数据略有变化的情况下产生最佳的性能？同样的预训练嵌入在多个神经网络体系结构上是否表现良好？未登录词的归罪策略会影响可靠性吗？在本文中，我们引入了两个新的度量来理解词嵌入的下游可靠性。我们发现，单词嵌入的下游可靠性取决于多个因素，包括评估指标、对词汇表外单词的处理以及嵌入是否经过微调。
While pre-trained word embeddings have been shown to improve the performance of downstream tasks, many questions remain regarding their reliability: Do the same pre-trained word embeddings result in the best performance with slight changes to the training data? Do the same pre-trained embeddings perform well with multiple neural network architectures? Do imputation strategies for unknown words impact reliability? In this paper, we introduce two new metrics to understand the downstream reliability of word embeddings. We find that downstream reliability of word embeddings depends on multiple factors, including, the evaluation metric, the handling of out-of-vocabulary words, and whether the embeddings are fine-tuned.