Attention-Focused Adversarial Training for Robust Temporal Reasoning
Attention-Focused Adversarial Training for Robust Temporal Reasoning
复制标题
DOI：
--
复制
发表时间：
2022
期刊：
影响因子：
--
通讯作者：
L. Pereira;Kevin Duh;Fei Cheng;Masayuki Asahara;Ichiro Kobayashi
中科院分区：
文献类型：
--
作者：
L. Pereira;Kevin Duh;Fei Cheng;Masayuki Asahara;Ichiro Kobayashi
我们提出了一种增强的对抗性训练算法，用于微调基于transformer的语言模型（即，RoBERTA）并将其应用于时间推理任务。目前NLP的对抗性训练方法仅将对抗性扰动添加到嵌入层，忽略了模型的其他层，这可能会限制对抗性训练的泛化能力。相反，我们的算法搜索层的最佳组合来添加对抗性扰动。我们将对抗扰动添加到模型层的多个隐藏状态或注意力表示中。在我们的实验中，将扰动添加到注意表征中表现最好。我们的模型可以提高性能的时间推理基准，并建立新的国家的最先进的结果。
We propose an enhanced adversarial training algorithm for fine-tuning transformer-based language models (i.e., RoBERTa) and apply it to the temporal reasoning task. Current adversarial training approaches for NLP add the adversarial perturbation only to the embedding layer, ignoring the other layers of the model, which might limit the generalization power of adversarial training. Instead, our algorithm searches for the best combination of layers to add the adversarial perturbation. We add the adversarial perturbation to multiple hidden states or attention representations of the model layers. Adding the perturbation to the attention representations performed best in our experiments. Our model can improve performance on several temporal reasoning benchmarks, and establishes new state-of-the-art results.