Embedding Compression with Isotropic Iterative Quantization
Embedding Compression with Isotropic Iterative Quantization
复制标题
DOI：
10.1609/aaai.v34i05.6350
复制
发表时间：
2020-01
期刊：
影响因子：
--
通讯作者：
Siyu Liao;Jie Chen;Yanzhi Wang;Qinru Qiu;Bo Yuan
中科院分区：
文献类型：
--
作者：
Siyu Liao;Jie Chen;Yanzhi Wang;Qinru Qiu;Bo Yuan
单词的连续表示是基于深度学习的 NLP 模型的标准组件。然而，表示大量词汇需要大量内存，这可能会导致问题，特别是在资源有限的平台上。因此，在本文中，我们提出了一种各向同性迭代量化（IIQ）方法，将嵌入向量压缩为二进制向量，利用图像检索中成熟的迭代量化技术，同时满足基于 PMI 的模型所需的各向同性。预训练嵌入（即 GloVe 和 HDC）的实验表明，与原始实值嵌入向量相比，压缩比提高了 30 倍以上，性能相当，有时甚至有所提高。
Continuous representation of words is a standard component in deep learning-based NLP models. However, representing a large vocabulary requires significant memory, which can cause problems, particularly on resource-constrained platforms. Therefore, in this paper we propose an isotropic iterative quantization (IIQ) approach for compressing embedding vectors into binary ones, leveraging the iterative quantization technique well established for image retrieval, while satisfying the desired isotropic property of PMI based models. Experiments with pre-trained embeddings (i.e., GloVe and HDC) demonstrate a more than thirty-fold compression ratio with comparable and sometimes even improved performance over the original real-valued embedding vectors.