SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
复制标题
DOI：
10.1162/tacl_a_00453
复制
发表时间：
2021-11
影响因子：
10.9
通讯作者：
Philippe Laban;Tobias Schnabel;Paul N. Bennett;Marti A. Hearst
中科院分区：
文献类型：
--
作者：
Philippe Laban;Tobias Schnabel;Paul N. Bennett;Marti A. Hearst
在摘要领域，摘要的一个关键要求是与输入文档事实上一致。之前的工作发现，自然语言推理（NLI）模型在应用于不一致检测时表现不具有竞争力。在这项工作中，我们重新审视了使用 NLI 进行不一致检测，发现过去的工作存在 NLI 数据集（句子级别）和不一致检测（文档级别）之间的输入粒度不匹配的问题。我们提供了一种名为 SummaCConv 的高效且轻量级的方法，该方法通过将文档分割成句子单元并聚合句子对之间的分数，使 NLI 模型能够成功地用于此任务。我们还引入了一个名为 SummaC（摘要一致性）的新基准，它由六个大型不一致检测数据集组成。在此数据集上，SummaCConv 获得了最先进的结果，平衡准确率为 74.4%，与之前的工作相比提高了 5%。
In the summarization domain, a key requirement for summaries is to be factually consistent with the input document. Previous work has found that natural language inference (NLI) models do not perform competitively when applied to inconsistency detection. In this work, we revisit the use of NLI for inconsistency detection, finding that past work suffered from a mismatch in input granularity between NLI datasets (sentence-level), and inconsistency detection (document level). We provide a highly effective and light-weight method called SummaCConv that enables NLI models to be successfully used for this task by segmenting documents into sentence units and aggregating scores between pairs of sentences. We furthermore introduce a new benchmark called SummaC (Summary Consistency) which consists of six large inconsistency detection datasets. On this dataset, SummaCConv obtains state-of-the-art results with a balanced accuracy of 74.4%, a 5% improvement compared with prior work.