SueNes: A Weakly Supervised Approach to Evaluating Single-Document Summarization via Negative Sampling
SueNes: A Weakly Supervised Approach to Evaluating Single-Document Summarization via Negative Sampling
复制标题
DOI：
10.18653/v1/2022.naacl-main.175
复制
发表时间：
2020-05
期刊：
影响因子：
--
通讯作者：
F. Bao;Hebi Li;Ge Luo;Minghui Qiu;Yinfei Yang;Youbiao He;Cen Chen
中科院分区：
文献类型：
--
作者：
F. Bao;Hebi Li;Ge Luo;Minghui Qiu;Yinfei Yang;Youbiao He;Cen Chen
典型的自动摘要评价指标，如ROUGE，侧重于词汇相似性，不能很好地捕捉语义和语言质量，需要一个参考摘要，这是昂贵的获得。最近，已经有越来越多的努力来减轻这两个缺点中的一个或两个。在本文中，我们提出了一个概念验证的研究，弱监督摘要评估方法，没有参考摘要的存在。现有摘要数据集中的大量数据通过将文档与损坏的参考摘要配对来进行转换以进行训练。在跨域测试中，我们的策略优于基线，具有很好的改进效果，并且在衡量所有指标的语言质量方面表现出很大的优势。
Canonical automatic summary evaluation metrics, such as ROUGE, focus on lexical similarity which cannot well capture semantics nor linguistic quality and require a reference summary which is costly to obtain. Recently, there have been a growing number of efforts to alleviate either or both of the two drawbacks. In this paper, we present a proof-of-concept study to a weakly supervised summary evaluation approach without the presence of reference summaries. Massive data in existing summarization datasets are transformed for training by pairing documents with corrupted reference summaries. In cross-domain tests, our strategy outperforms baselines with promising improvements, and show a great advantage in gauging linguistic qualities over all metrics.