Extractive is not Faithful: An Investigation of Broad Unfaithfulness Problems in Extractive Summarization
Extractive is not Faithful: An Investigation of Broad Unfaithfulness Problems in Extractive Summarization
复制标题
DOI：
10.48550/arxiv.2209.03549
复制
发表时间：
2022-09
期刊：
影响因子：
--
通讯作者：
Shiyue Zhang;David Wan;Mohit Bansal
中科院分区：
文献类型：
--
作者：
Shiyue Zhang;David Wan;Mohit Bansal
在抽象摘要的背景下，不忠实摘要的问题已经被广泛讨论。尽管提取摘要不太容易出现抽象摘要常见的不忠实问题，但这是否意味着提取就等于忠实？事实证明答案是否定的。在这项工作中，我们定义了一种类型学，其中包含五种可能出现在提取摘要中的广泛不忠问题（包括和超越不蕴涵），包括不正确的共指、不完整的共指、不正确的话语、不完整的话语以及其他误导性信息。我们要求人类从 16 个不同的提取系统生成的 1600 个英文摘要中标记这些问题。我们发现 30% 的摘要至少存在这五个问题之一。为了自动检测这些问题，我们发现现有的 5 个摘要可信度评估指标与人类判断的相关性较差。为了解决这个问题，我们提出了一个新的指标 ExtEval，它是为检测不忠实的提取摘要而设计的，并且被证明具有最佳性能。我们希望我们的工作能够提高人们对提取概括中不忠实问题的认识，并有助于今后的工作评估和解决这些问题。
The problems of unfaithful summaries have been widely discussed under the context of abstractive summarization. Though extractive summarization is less prone to the common unfaithfulness issues of abstractive summaries, does that mean extractive is equal to faithful? Turns out that the answer is no. In this work, we define a typology with five types of broad unfaithfulness problems (including and beyond not-entailment) that can appear in extractive summaries, including incorrect coreference, incomplete coreference, incorrect discourse, incomplete discourse, as well as other misleading information. We ask humans to label these problems out of 1600 English summaries produced by 16 diverse extractive systems. We find that 30% of the summaries have at least one of the five issues. To automatically detect these problems, we find that 5 existing faithfulness evaluation metrics for summarization have poor correlations with human judgment. To remedy this, we propose a new metric, ExtEval, that is designed for detecting unfaithful extractive summaries and is shown to have the best performance. We hope our work can increase the awareness of unfaithfulness problems in extractive summarization and help future work to evaluate and resolve these issues.