Evaluating the Calibration of Knowledge Graph Embeddings for Trustworthy Link Prediction
Evaluating the Calibration of Knowledge Graph Embeddings for Trustworthy Link Prediction
复制标题
DOI：
10.18653/v1/2020.emnlp-main.667
复制
发表时间：
2020-04
期刊：
影响因子：
--
通讯作者：
Tara Safavi;Danai Koutra;E. Meij
中科院分区：
文献类型：
--
作者：
Tara Safavi;Danai Koutra;E. Meij
人们对知识图嵌入（KGE）模型预测的可信度知之甚少。在本文中，我们通过研究 KGE 模型的校准，或者它们输出反映预测知识图三元组预期正确性的置信度分数的程度，朝着这个方向迈出了第一步。我们首先在标准封闭世界假设（CWA）下进行评估，其中尚未出现在知识图中的预测三元组被认为是错误的，并表明现有的校准技术在这种常见但狭隘的假设下对于 KGE 是有效的。接下来，我们引入更现实但更具挑战性的开放世界假设（OWA），其中在获得真实标签之前，未观察到的预测不被认为是真或假。在这里，我们表明现有的校准技术在 OWA 下的效率远低于 CWA，并为这种差异提供了解释。最后，为了从从业者的角度激发 KGE 校准的效用，我们进行了人类与人工智能协作的独特案例研究，表明校准预测可以提高人类在知识图谱完成任务中的表现。
Little is known about the trustworthiness of predictions made by knowledge graph embedding (KGE) models. In this paper we take initial steps toward this direction by investigating the calibration of KGE models, or the extent to which they output confidence scores that reflect the expected correctness of predicted knowledge graph triples. We first conduct an evaluation under the standard closed-world assumption (CWA), in which predicted triples not already in the knowledge graph are considered false, and show that existing calibration techniques are effective for KGE under this common but narrow assumption. Next, we introduce the more realistic but challenging open-world assumption (OWA), in which unobserved predictions are not considered true or false until ground-truth labels are obtained. Here, we show that existing calibration techniques are much less effective under the OWA than the CWA, and provide explanations for this discrepancy. Finally, to motivate the utility of calibration for KGE from a practitioner's perspective, we conduct a unique case study of human-AI collaboration, showing that calibrated predictions can improve human performance in a knowledge graph completion task.