PaCo: Preconditions Attributed to Commonsense Knowledge
PaCo: Preconditions Attributed to Commonsense Knowledge
复制标题
DOI：
10.18653/v1/2022.findings-emnlp.505
复制
发表时间：
2021-04
期刊：
影响因子：
--
通讯作者：
Ehsan Qasemi;Filip Ilievski;Muhao Chen;Pedro A. Szekely
中科院分区：
文献类型：
--
作者：
Ehsan Qasemi;Filip Ilievski;Muhao Chen;Pedro A. Szekely
人类可以用常识知识的间接前提进行无缝推理。我们知道，玻璃杯是用来喝水的，除非玻璃杯被打破或水有毒。尽管最先进的（SOTA）语言模型（LM）在推断常识知识方面的表现令人印象深刻，但尚不清楚它们是否理解环境前提条件。为了解决这一差距，我们提出了一个新的挑战推理与间接的先决条件。我们收集了一个名为PaCo的数据集，由12400个用自然语言表达的常识性语句的前提条件组成。基于这个数据集，我们创建了三个典型的评估任务，并使用它们来检查现有LM理解情境前提条件的能力。我们的研究结果显示，在我们的任务中，机器和人类的表现之间存在10-30%的差距，这表明带有前提条件的推理是一个开放的挑战。
Humans can seamlessly reason with circumstantial preconditions of commonsense knowledge. We understand that a glass is used for drinking water, unless the glass is broken or the water is toxic. Despite state-of-the-art (SOTA) language models' (LMs) impressive performance on inferring commonsense knowledge, it is unclear whether they understand the circumstantial preconditions. To address this gap, we propose a novel challenge of reasoning with circumstantial preconditions. We collect a dataset, called PaCo, consisting of 12.4 thousand preconditions of commonsense statements expressed in natural language. Based on this dataset, we create three canonical evaluation tasks and use them to examine the capability of existing LMs to understand situational preconditions. Our results reveal a 10-30% gap between machine and human performance on our tasks, which shows that reasoning with preconditions is an open challenge.