Neural Word Embedding as Implicit Matrix Factorization
Neural Word Embedding as Implicit Matrix Factorization
复制标题
DOI：
--
复制
发表时间：
2014-12
期刊：
影响因子：
--
通讯作者：
Omer Levy;Yoav Goldberg
中科院分区：
文献类型：
--
作者：
Omer Levy;Yoav Goldberg
我们用负采样（SGNS）分析skip-gram，这是Mikolov等人提出的一种单词嵌入方法，并表明，它是隐式因式分解的词上下文矩阵，其细胞是逐点互信息（PMI）的相应的词和上下文对，由一个全局常数移动。我们发现，另一种嵌入方法，NCE，是隐式因式分解一个类似的矩阵，其中每个单元格是一个词的（移位）对数条件概率给定其上下文。我们发现，使用稀疏移位正PMI词上下文矩阵来表示单词，可以提高两个单词相似性任务和两个类比任务之一的结果。当首选密集低维向量时，使用SVD的精确因式分解可以实现至少与SGNS解决方案一样好的解决方案。在类比问题上，SGNS仍然上级SVD。我们推测，这源于SGNS的因式分解的加权性质。
We analyze skip-gram with negative-sampling (SGNS), a word embedding method introduced by Mikolov et al., and show that it is implicitly factorizing a word-context matrix, whose cells are the pointwise mutual information (PMI) of the respective word and context pairs, shifted by a global constant. We find that another embedding method, NCE, is implicitly factorizing a similar matrix, where each cell is the (shifted) log conditional probability of a word given its context. We show that using a sparse Shifted Positive PMI word-context matrix to represent words improves results on two word similarity tasks and one of two analogy tasks. When dense low-dimensional vectors are preferred, exact factorization with SVD can achieve solutions that are at least as good as SGNS's solutions for word similarity tasks. On analogy questions SGNS remains superior to SVD. We conjecture that this stems from the weighted nature of SGNS's factorization.