How Can We Know What Language Models Know?
How Can We Know What Language Models Know?
复制标题
DOI：
10.1162/tacl_a_00324
复制
发表时间：
2020-01-01
影响因子：
10.9
通讯作者：
Neubig, Graham
中科院分区：
文献类型：
--
作者：
Jiang, Zhengbao;Xu, Frank F.;Neubig, Graham
最近的工作提出了有趣的结果，以检查语言模型（LMS）中所包含的知识，通过将LM填充在诸如“ Obama是_ OBAMA _ BY PROVERCE'”之类的提示中。这些提示通常是手动创建的，并且很可能很可能子 - 最佳；例如，“奥巴马为A_”这样的提示可能会更准确地预测正确的职业。因此，如果提示不适当，我们可能无法检索LM确实知道的事实，因此任何给定的提示都只能提供对LM中包含的知识的下限估计。在本文中，我们试图通过自动发现在此查询过程中使用更好的提示来更准确地估算LMS中所含的知识。具体而言，我们提出了基于采矿的方法和基于释义的方法，以自动生成高质量和不同的提示，以及合奏方法，以结合不同提示的答案。从LMS中提取关系知识的喇嘛基准进行的广泛实验表明，我们的方法可以将准确性从31.1％提高到39.6％，从而在LMS所知道的方面具有更严格的下限。我们已经发布了代码和所得的LM提示和查询存档（LPAQA），网址为https://github.com/jzbjyb/lpaqa。
Recent work has presented intriguing results examining the knowledge contained in language models (LMs) by having the LM fill in the blanks of prompts such as "Obama is a _ by profession''. These prompts are usually manually created, and quite possibly sub-optimal; another prompt such as "Obama worked as a_'' may result in more accurately predicting the correct profession. Because of this, given an inappropriate prompt, we might fail to retrieve facts that the LM does know, and thus any given prompt only provides a lower bound estimate of the knowledge contained in an LM. In this paper, we attempt to more accurately estimate the knowledge contained in LMs by automatically discovering better prompts to use in this querying process. Specifically, we propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. Extensive experiments on the LAMA benchmark for extracting relational knowledge from LMs demonstrate that our methods can improve accuracy from 31.1% to 39.6%, providing a tighter lower bound on what LMs know. We have released the code and the resulting LM Prompt And Query Archive (LPAQA) at https://github.com/jzbjyb/LPAQA.