On the Importance of Delexicalization for Fact Verification
On the Importance of Delexicalization for Fact Verification
复制标题
论去词汇化对于事实验证的重要性
DOI：
--
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
M. Surdeanu
中科院分区：
文献类型：
--
作者：
Sandeep Suntwal;Mitch Paul Mithun;Rebecca Sharp;M. Surdeanu
尽管神经网络在许多NLP任务中产生最先进的性能，但它们通常从词汇信息中学习，词汇信息可能会在域之间传递不佳。在这里，我们调查了模型在学习和做出预测的同时，特别是在识别文本构成（RTE）任务中分配给数据的各个方面的重要性。通过检查模型分配的注意力重量，我们确认大多数权重分配给了名词短语。为了减轻对词汇化信息的依赖，我们尝试了两种掩盖策略。首先，我们用其相应的语义标签替换了指定的实体，以及一个唯一的标识符，以指示索赔和证据之间的词汇叠加。其次，我们同样替换了句子中的其他单词类（名词，动词，形容词和副词），并用它们的超级感标签（Ciaramita and Johnson，2003）。我们的结果表明，尽管在域中数据集上的性能与在完全词汇化数据中训练的模型相当，但在域中测试时，它会大大提高。例如，在蒙面的假新闻挑战（Pomerleau and Rao，2017年）中训练的最先进的RTE模型的性能，并对事实提取和验证进行了评估（Thorne等，2018）数据通过超过与完全词汇化模型相比，准确分数为10％。
While neural networks produce state-of-the-art performance in many NLP tasks, they generally learn from lexical information, which may transfer poorly between domains. Here, we investigate the importance that a model assigns to various aspects of data while learning and making predictions, specifically, in a recognizing textual entailment (RTE) task. By inspecting the attention weights assigned by the model, we confirm that most of the weights are assigned to noun phrases. To mitigate this dependence on lexicalized information, we experiment with two strategies of masking. First, we replace named entities with their corresponding semantic tags along with a unique identifier to indicate lexical overlap between claim and evidence. Second, we similarly replace other word classes in the sentence (nouns, verbs, adjectives, and adverbs) with their super sense tags (Ciaramita and Johnson, 2003). Our results show that, while performance on the in-domain dataset remains on par with that of the model trained on fully lexicalized data, it improves considerably when tested out of domain. For example, the performance of a state-of-the-art RTE model trained on the masked Fake News Challenge (Pomerleau and Rao, 2017) data and evaluated on Fact Extraction and Verification (Thorne et al., 2018) data improved by over 10% in accuracy score compared to the fully lexicalized model.