Distilling Semantic Concept Embeddings from Contrastively Fine-Tuned Language Models
Distilling Semantic Concept Embeddings from Contrastively Fine-Tuned Language Models
复制标题
DOI：
10.1145/3539618.3591667
复制
发表时间：
2023-05
期刊：
影响因子：
--
通讯作者：
N. Li;Hanane Kteich;Zied Bouraoui;Steven Schockaert
中科院分区：
文献类型：
--
作者：
N. Li;Hanane Kteich;Zied Bouraoui;Steven Schockaert
捕捉概念含义的学习向量仍然是一个根本性挑战。也许有些令人惊讶的是，到目前为止，预先训练的语言模型只对这种概念嵌入的质量进行了适度的改进。当前使用语言模型的策略通常通过在一些语料库中对其提及的上下文化表示进行平均来表示概念。由于至少两个原因，这可能是次优的。首先，语境化的词向量具有不寻常的几何形状，这阻碍了下游任务。其次，概念嵌入应该捕捉概念的语义属性，而上下文化的词向量还受到其他因素的影响。为了解决这些问题，我们提出了两种对比学习策略，基于这样的观点，即每当两个句子揭示相似的属性时，相应的上下文向量也应该是相似的。一种策略是完全无监督的，从上下文化的词嵌入的邻域结构估计在句子中表达的属性。第二种策略依赖于来自ConceptNet的远程监督信号。我们的实验结果表明，所产生的向量大大优于现有的概念嵌入在预测概念的语义属性，与ConceptNet为基础的战略取得了最好的结果。这些发现进一步证实了聚类任务和本体完成的下游任务。
Learning vectors that capture the meaning of concepts remains a fundamental challenge. Somewhat surprisingly, perhaps, pre-trained language models have thus far only enabled modest improvements to the quality of such concept embeddings. Current strategies for using language models typically represent a concept by averaging the contextualised representations of its mentions in some corpus. This is potentially sub-optimal for at least two reasons. First, contextualised word vectors have an unusual geometry, which hampers downstream tasks. Second, concept embeddings should capture the semantic properties of concepts, whereas contextualised word vectors are also affected by other factors. To address these issues, we propose two contrastive learning strategies, based on the view that whenever two sentences reveal similar properties, the corresponding contextualised vectors should also be similar. One strategy is fully unsupervised, estimating the properties which are expressed in a sentence from the neighbourhood structure of the contextualised word embeddings. The second strategy instead relies on a distant supervision signal from ConceptNet. Our experimental results show that the resulting vectors substantially outperform existing concept embeddings in predicting the semantic properties of concepts, with the ConceptNet-based strategy achieving the best results. These findings are furthermore confirmed in a clustering task and in the downstream task of ontology completion.