SVD-Softmax: Fast Softmax Approximation on Large Vocabulary Neural Networks
SVD-Softmax: Fast Softmax Approximation on Large Vocabulary Neural Networks
复制标题
SVD-Softmax：大词汇量神经网络的快速 Softmax 逼近
DOI：
--
复制
发表时间：
2017
期刊：
影响因子：
--
通讯作者：
Wonyong Sung
中科院分区：
文献类型：
--
作者：
Kyuhong Shim;Minjae Lee;Iksoo Choi;Yoonho Boo;Wonyong Sung
我们提出了一种快速逼近方法的softmax函数与一个非常大的词汇使用奇异值分解（SVD）。SVD-softmax的目标是在神经网络语言模型的推理过程中快速准确地估计最可能的单词的概率。该方法通过奇异值分解对输出向量计算中使用的权矩阵进行变换。通过对大部分单词使用少量大的奇异值和相应的元素，可以仅用权重矩阵的一小部分来估计每个单词的近似概率。我们将该技术应用于语言建模和神经机器翻译，并提出了一个良好的近似准则。该算法只需要大约20%的算术运算的800 K词汇的情况下，并显示了超过三倍的GPU上的加速。
We propose a fast approximation method of a softmax function with a very large vocabulary using singular value decomposition (SVD). SVD-softmax targets fast and accurate probability estimation of the topmost probable words during inference of neural network language models. The proposed method transforms the weight matrix used in the calculation of the output vector by using SVD. The approximate probability of each word can be estimated with only a small part of the weight matrix by using a few large singular values and the corresponding elements for most of the words. We applied the technique to language modeling and neural machine translation and present a guideline for good approximation. The algorithm requires only approximately 20\% of arithmetic operations for an 800K vocabulary case and shows more than a three-fold speedup on a GPU.