Contrastive Learning of Sentence Representations
Contrastive Learning of Sentence Representations
复制标题
DOI：
--
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Hefei Qiu;Wei Ding;Ping Chen
中科院分区：
文献类型：
--
作者：
Hefei Qiu;Wei Ding;Ping Chen
学习能够捕捉到丰富语义的句子表征对于许多NLP任务来说是至关重要的。BERT等预先训练的语言模型在NLP中取得了巨大的成功，但直接从这些模型中提取的句子嵌入在没有微调的情况下表现不佳。我们提出了句子表征的对比学习（CLSR），这是一种新的方法，它应用对比学习来学习预训练语言模型之上的通用句子表征。CLSR利用两个句子的语义相似性来构造用于对比学习的正例。通过适当的池化策略从这些模型中获取句子嵌入，保留了预训练模型捕获的语义信息。编码器后跟线性投影，将这些嵌入作为输入，并在对比目标下进行训练。为了评估CLSR的性能，我们对一系列预先训练的语言模型及其变体进行了一系列语义上下文相似性任务的实验。结果表明，CLSR获得了显着的性能改善现有的SOTA语言模型。
Learning sentence representations which capture rich semantic meanings has been crucial for many NLP tasks. Pre-trained language models such as BERT have achieved great success in NLP, but sentence embeddings extracted directly from these models do not perform well without fine-tuning. We propose Contrastive Learning of Sentence Representations (CLSR), a novel approach which applies contrastive learning to learn universal sentence representations on top of pre-trained language models. CLSR utilizes semantic similarity of two sentences to construct positive instance for contrastive learning. Semantic information that has been captured by the pre-trained models is kept by getting sentence embeddings from these models with proper pooling strategy. An encoder followed by a linear projection takes these embeddings as inputs and is trained under a contrastive objective. To evaluate the performance of CLSR, we run experiments on a range of pre-trained language models and their variants on a series of Semantic Contextual Similarity tasks. Results show that CLSR gains significant performance improvements over existing SOTA language models.