Revisiting Additive Compositionality: AND, OR and NOT Operations with Word Embeddings
Revisiting Additive Compositionality: AND, OR and NOT Operations with Word Embeddings
复制标题
DOI：
--
复制
发表时间：
2021-05
期刊：
影响因子：
--
通讯作者：
Masahiro Naito;Sho Yokoi;Geewook Kim;Hidetoshi Shimodaira
中科院分区：
文献类型：
--
作者：
Masahiro Naito;Sho Yokoi;Geewook Kim;Hidetoshi Shimodaira
众所周知，典型的词嵌入001方法具有可以通过将嵌入003相加来组合含义002的属性（加法组合性）。人们提出了几种理论来解释加性复合性，但仍然存在以下问题：（1）这些理论的假设不适用于实际的词嵌入。(ii)二进制加法组合可以被看作是词义的AND运算，但还不清楚其他运算，如OR和NOT，如何通过012嵌入来计算。我们解决这些问题的013频率加权为中心的想法在其核心。014该方法弥合了实际015单词嵌入和理论016关于加法组合性的假设之间的差距，作为对017（i）的回答。本文还给出了用线性运算取意义的018或非--词嵌入的019作为问题（ii）的答案的方法。此外，我们通过实验证实了AND运算的精度，即，我们的后处理方法可以改善普通的022加法组合性（在前100位精度中提高了3.5倍），并且可以正确地执行OR和NOT运算。我们还确认所提出的方法对于BERT是有效的。028
It is well-known that typical word embedding 001 methods have the property that the meaning 002 can be composed by adding up the embeddings 003 (additive compositionality). Several theories 004 have been proposed to explain additive compo- 005 sitionality, but the following problems remain: 006 (i) The assumptions of those theories do not 007 hold for the practical word embedding. (ii) Or- 008 dinary additive compositionality can be seen 009 as an AND operation of word meanings, but 010 it is not well understood how other operations, 011 such as OR and NOT, can be computed by the 012 embeddings. We address these issues by the 013 idea of frequency-weighted centering at its core. 014 This method bridges the gap between practical 015 word embedding and the assumption of theory 016 about additive compositionality as an answer to 017 (i). This paper also gives a method for taking 018 OR or NOT of the meaning by linear opera- 019 tion of word embedding as an answer to (ii). 020 Moreover, we confirm experimentally that the 021 accuracy of AND operation, i.e., the ordinary 022 additive compositionality, can be improved by 023 our post-processing method (3.5x improvement 024 in top-100 accuracy) and that OR and NOT op- 025 erations can be performed correctly. We also 026 confirm that the proposed method is effective 027 for BERT. 028