The strange geometry of skip-gram with negative sampling
The strange geometry of skip-gram with negative sampling
复制标题
DOI：
10.18653/v1/d17-1308
复制
发表时间：
2017-09
期刊：
影响因子：
--
通讯作者：
David Mimno;Laure Thompson
中科院分区：
文献类型：
--
作者：
David Mimno;Laure Thompson
尽管它们无处不在，但使用skip-gram negative sampling（SGNS）训练的词嵌入仍然知之甚少。我们发现，向量的位置不是简单地由语义相似性决定的，而是占据一个狭窄的圆锥体，与上下文向量截然相反。我们表明，这种几何浓度取决于比例的正面和负面的例子，它既不是理论上也不是经验上固有的相关嵌入算法。
Despite their ubiquity, word embeddings trained with skip-gram negative sampling (SGNS) remain poorly understood. We find that vector positions are not simply determined by semantic similarity, but rather occupy a narrow cone, diametrically opposed to the context vectors. We show that this geometric concentration depends on the ratio of positive to negative examples, and that it is neither theoretically nor empirically inherent in related embedding algorithms.