Comparison and Combination of Sentence Embeddings Derived from Different Supervision Signals
Comparison and Combination of Sentence Embeddings Derived from Different Supervision Signals
复制标题
DOI：
10.18653/v1/2022.starsem-1.12
复制
发表时间：
2022-02
期刊：
影响因子：
--
通讯作者：
Hayato Tsukagoshi;Ryohei Sasano;Koichi Takeda
中科院分区：
文献类型：
--
作者：
Hayato Tsukagoshi;Ryohei Sasano;Koichi Takeda
句子嵌入方法有许多成功的应用。然而，人们还没有很好地理解，根据监督信号，在生成的句子嵌入中捕获了哪些属性。本文研究了两种结构和任务相似的句子嵌入方法：一种是基于自然语言推理任务的微调预训练语言模型，另一种是基于定义句的词预测任务的微调预训练语言模型，并研究了它们的性质。具体来说，我们比较了他们在语义文本相似（STS）任务上的表现，使用从两个角度(1)句子源和2)句子对表面相似度划分的STS数据集，并比较了他们在下游任务和探测任务上的表现。此外，我们试图将这两种方法结合起来，并证明结合这两种方法在无监督STS任务和下游任务上比各自的方法产生更好的性能。
There have been many successful applications of sentence embedding methods.However, it has not been well understood what properties are captured in the resulting sentence embeddings depending on the supervision signals.In this paper, we focus on two types of sentence embedding methods with similar architectures and tasks: one fine-tunes pre-trained language models on the natural language inference task, and the other fine-tunes pre-trained language models on word prediction task from its definition sentence, and investigate their properties.Specifically, we compare their performances on semantic textual similarity (STS) tasks using STS datasets partitioned from two perspectives: 1) sentence source and 2) superficial similarity of the sentence pairs, and compare their performances on the downstream and probing tasks.Furthermore, we attempt to combine the two methods and demonstrate that combining the two methods yields substantially better performance than the respective methods on unsupervised STS tasks and downstream tasks.