SNaC: Coherence Error Detection for Narrative Summarization
SNaC: Coherence Error Detection for Narrative Summarization
复制标题
DOI：
10.48550/arxiv.2205.09641
复制
发表时间：
2022-05
期刊：
影响因子：
--
通讯作者：
Tanya Goyal;Junyi Jessy Li;Greg Durrett
中科院分区：
文献类型：
--
作者：
Tanya Goyal;Junyi Jessy Li;Greg Durrett
由于缺乏适当的评价框架，在总结长篇案文方面的进展受到阻碍。适当涵盖案文各方面的长篇摘要还必须提供连贯的叙述，但目前的自动和人工评价方法无法找出连贯性方面的差距。在这项工作中，我们介绍SNaC，一个叙事连贯性评价框架，用于细粒度的长摘要注释。我们开发了一个分类法的连贯性错误生成的叙述性摘要，并收集跨度级注释6.6k句子在150本书和电影摘要。我们的工作提供了第一个表征的一致性错误所产生的国家的最先进的摘要模型和一个协议，用于引起一致性的判断，从群众工作者。此外，我们表明，收集的注释使我们能够基准过去的工作在连贯性建模和训练一个强大的分类器自动本地化生成的摘要中的连贯性错误。最后，我们的SNaC框架可以支持长文档摘要和一致性评估的未来工作，包括改进的摘要建模和事后摘要校正。
Progress in summarizing long texts is inhibited by the lack of appropriate evaluation frameworks. A long summary that appropriately covers the facets of that text must also present a coherent narrative, but current automatic and human evaluation methods fail to identify gaps in coherence. In this work, we introduce SNaC, a narrative coherence evaluation framework for fine-grained annotations of long summaries. We develop a taxonomy of coherence errors in generated narrative summaries and collect span-level annotations for 6.6k sentences across 150 book and movie summaries. Our work provides the first characterization of coherence errors generated by state-of-the-art summarization models and a protocol for eliciting coherence judgments from crowdworkers. Furthermore, we show that the collected annotations allow us to benchmark past work in coherence modeling and train a strong classifier for automatically localizing coherence errors in generated summaries. Finally, our SNaC framework can support future work in long document summarization and coherence evaluation, including improved summarization modeling and post-hoc summary correction.