Knowledge Enhanced Masked Language Model for Stance Detection
Knowledge Enhanced Masked Language Model for Stance Detection
复制标题
DOI：
10.18653/v1/2021.naacl-main.376
复制
发表时间：
2021-06
期刊：
影响因子：
--
通讯作者：
Kornraphop Kawintiranon;L. Singh
中科院分区：
文献类型：
--
作者：
Kornraphop Kawintiranon;L. Singh
在Twitter上检测立场尤其具有挑战性，因为每条推文的长度很短，新术语和主题标签的连续造币以及句子结构偏离标准散文。使用大规模内域数据的微调语言模型已被证明是许多NLP任务（包括立场检测）的新最新技术。在本文中，我们提出了一种基于BERT的新型微调方法，可以增强掩盖语言模型以进行立场检测。我们建议使用加权log-odds-ratio识别具有高立场可区分性的单词，然后建模关注这些单词的注意机制，而不是随机令牌掩盖。我们表明，我们提出的方法在Twitter数据上的立场检测表现优于2020年美国总统大选的立场检测状态。
Detecting stance on Twitter is especially challenging because of the short length of each tweet, the continuous coinage of new terminology and hashtags, and the deviation of sentence structure from standard prose. Fine-tuned language models using large-scale in-domain data have been shown to be the new state-of-the-art for many NLP tasks, including stance detection. In this paper, we propose a novel BERT-based fine-tuning method that enhances the masked language model for stance detection. Instead of random token masking, we propose using a weighted log-odds-ratio to identify words with high stance distinguishability and then model an attention mechanism that focuses on these words. We show that our proposed approach outperforms the state of the art for stance detection on Twitter data about the 2020 US Presidential election.