Spying on Your Neighbors: Fine-grained Probing of Contextual Embeddings for Information about Surrounding Words
Spying on Your Neighbors: Fine-grained Probing of Contextual Embeddings for Information about Surrounding Words
复制标题
DOI：
10.18653/v1/2020.acl-main.434
复制
发表时间：
2020-05
影响因子：
1.5
通讯作者：
Josef Klafka;Allyson Ettinger
中科院分区：
文献类型：
--
作者：
Josef Klafka;Allyson Ettinger
尽管使用上下文词嵌入的模型在许多 NLP 任务上取得了最先进的结果，但人们对这些嵌入到底编码了哪些关于它们被理解为反映的上下文词的信息知之甚少。为了解决这个问题，我们引入了一套探测任务，可以对上下文嵌入进行细粒度测试，以对周围单词的信息进行编码。我们应用这些任务来检查流行的 BERT、ELMo 和 GPT 上下文编码器，发现我们测试的每种信息类型确实被编码为跨 token 的上下文信息，通常具有近乎完美的可恢复性，但编码器在将哪些特征分配给哪些 token、分布的细微差别以及每个特征的编码对距离的鲁棒性方面有所不同。我们讨论这些结果对不同类型的模型在构建标记嵌入时如何分解和优先考虑单词级上下文信息的影响。
Although models using contextual word embeddings have achieved state-of-the-art results on a host of NLP tasks, little is known about exactly what information these embeddings encode about the context words that they are understood to reflect. To address this question, we introduce a suite of probing tasks that enable fine-grained testing of contextual embeddings for encoding of information about surrounding words. We apply these tasks to examine the popular BERT, ELMo and GPT contextual encoders, and find that each of our tested information types is indeed encoded as contextual information across tokens, often with near-perfect recoverability—but the encoders vary in which features they distribute to which tokens, how nuanced their distributions are, and how robust the encoding of each feature is to distance. We discuss implications of these results for how different types of models break down and prioritize word-level context information when constructing token embeddings.