ISD-QA: Iterative Distillation of Commonsense Knowledge from General Language Models for Unsupervised Question Answering
ISD-QA: Iterative Distillation of Commonsense Knowledge from General Language Models for Unsupervised Question Answering
复制标题
DOI：
10.1109/icpr56361.2022.9956441
复制
发表时间：
2022-08
期刊：
影响因子：
--
通讯作者：
Priyadharsini Ramamurthy;Sathyanarayanan N. Aakur
中科院分区：
文献类型：
--
作者：
Priyadharsini Ramamurthy;Sathyanarayanan N. Aakur
常识性问题的回答主要是通过有监督的迁移学习来解决的，在这种学习中，根据大量数据预先训练的语言模型被用作起点。虽然成功，但该方法需要大量标记的问题-答案对，随着场景或任务(如常识QA)的复杂性增加，所需的数据量越来越大。在本文中，我们假设语言模型的大规模预训练编码了必要的常识知识，以便在没有标记数据的情况下回答常见问题。我们提出了一种新的问答框架--迭代自精馏问答(ISD-QA)，该框架提取语言模型大规模预训练过程中编码的“暗知识”，为常识性问题的回答提供监督。我们表明，该方法可以通过无监督的方式从语言模型中提取知识，来训练用于常识性问题回答的常见神经QA模型。在没有花哨的情况下，我们获得了平均68%的性能完全监督的QA模型，而不需要标记的训练数据。在三个公共基准测试(OpenBookQA、HellaSWAG和CommonsenseQA)上的大量实验表明了该方法的有效性。
Commonsense question answering has primarily been tackled through supervised transfer learning, where a language model pre-trained on large amounts of data is used as the starting point. While successful, the approach requires large amounts of labeled question-answer pairs, with increasingly larger amounts of data required as the complexity of scenarios or tasks such as commonsense QA increases. In this paper, we hypothesize that large-scale pre-training of language models encodes the necessary commonsense knowledge to answer common questions in context without labeled data. We propose a novel framework called Iterative Self Distillation for QA (ISD-QA), which extracts the "dark knowledge" encoded during largescale pre-training of language models to provide supervision for commonsense question answering. We show that the approach can be used to train common neural QA models for commonsense question answering by distilling knowledge from language models in an unsupervised manner. With no bells and whistles, we achieve an average of 68% of the performance of fully supervised QA models while requiring no labeled training data. Extensive experiments on three public benchmarks (OpenBookQA, HellaSWAG, and CommonsenseQA) show the effectiveness of the proposed approach.