SUBSUME: A Dataset for Subjective Summary Extraction from Wikipedia Documents
SUBSUME: A Dataset for Subjective Summary Extraction from Wikipedia Documents
复制标题
DOI：
10.18653/v1/2021.newsum-1.14
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Nishant Yadav;Matteo Brucato;Anna Fariha;Oscar Youngquist;J. Killingback;A. Meliou;Peter J. Haas
中科院分区：
文献类型：
--
作者：
Nishant Yadav;Matteo Brucato;Anna Fariha;Oscar Youngquist;J. Killingback;A. Meliou;Peter J. Haas
许多应用需要生成适合于用户信息需求的摘要，即，他们的意图当查询解释是主观的时，通过显式用户查询来表达意图的方法就不合适了。存在用于具有客观意图的摘要的若干数据集，其中，对于每个文档和意图（例如，“weather”），则单个摘要足以用于所有用户。然而，不存在用于主观意图的数据集（例如，“有趣的地方”），其中不同的用户将提供不同的摘要。我们提出了SUBSUME，第一个用于评估主观总结提取系统的数据集。SUBSUME包含超过48个维基百科页面的2,200个（文档，意图，摘要）三元组，其中10个意图具有不同的主观性，由103个人在Mechanical Turk. We统计表明，SUBSUME中的意图在主观性上系统地变化。为了表明SUBSUME的有用性，我们探索了一系列用于主观提取摘要的基线算法，并表明（i）正如预期的那样，基于示例的方法比基于查询的方法更好地捕获主观意图，以及（ii）基线算法有足够的改进空间，从而激励对这个具有挑战性的问题进行进一步的研究。
Many applications require generation of summaries tailored to the user’s information needs, i.e., their intent. Methods that express intent via explicit user queries fall short when query interpretation is subjective. Several datasets exist for summarization with objective intents where, for each document and intent (e.g., “weather”), a single summary suffices for all users. No datasets exist, however, for subjective intents (e.g., “interesting places”) where different users will provide different summaries. We present SUBSUME, the first dataset for evaluation of SUBjective SUMmary Extraction systems. SUBSUME contains 2,200 (document, intent, summary) triplets over 48 Wikipedia pages, with ten intents of varying subjectivity, provided by 103 individuals over Mechanical Turk. We demonstrate statistically that the intents in SUBSUME vary systematically in subjectivity. To indicate SUBSUME’s usefulness, we explore a collection of baseline algorithms for subjective extractive summarization and show that (i) as expected, example-based approaches better capture subjective intents than query-based ones, and (ii) there is ample scope for improving upon the baseline algorithms, thereby motivating further research on this challenging problem.