Hurdles to Progress in Long-form Question Answering
Hurdles to Progress in Long-form Question Answering
复制标题
DOI：
10.18653/v1/2021.naacl-main.393
复制
发表时间：
2021-03
期刊：
影响因子：
--
通讯作者：
Kalpesh Krishna;Aurko Roy;Mohit Iyyer
中科院分区：
文献类型：
--
作者：
Kalpesh Krishna;Aurko Roy;Mohit Iyyer
长格式问答(LFQA)任务涉及检索与给定问题相关的文档，并使用它们生成段落长度的答案。虽然最近提出了许多用于LFQA的模型，但在本文中，我们表明任务公式在评估和数据集创建方面提出了根本挑战，目前阻碍了有意义的建模进展。为了展示这些挑战，我们首先设计了一个新的系统，它依靠稀疏注意和对比检索器学习来在ELI5 LFQA数据集上实现最先进的性能。虽然我们的系统在公共排行榜上名列前茅，但详细的分析揭示了几个令人不安的趋势：(1)我们系统生成的答案实际上并不基于它检索的文档；(2)ELI5包含显著的训练/验证重叠，因为至少81%的ELI5验证问题以释义的形式出现在训练集中；(3)Rouge-L不是生成答案质量的信息性度量，很容易被玩弄；以及(4)用于其他文本生成任务的人工评估对于LFQA是不可靠的。我们提供了缓解这些问题的建议，我们希望这些建议将导致更严格的LFQA研究和未来有意义的进展。
The task of long-form question answering (LFQA) involves retrieving documents relevant to a given question and using them to generate a paragraph-length answer. While many models have recently been proposed for LFQA, we show in this paper that the task formulation raises fundamental challenges regarding evaluation and dataset creation that currently preclude meaningful modeling progress. To demonstrate these challenges, we first design a new system that relies on sparse attention and contrastive retriever learning to achieve state-of-the-art performance on the ELI5 LFQA dataset. While our system tops the public leaderboard, a detailed analysis reveals several troubling trends: (1) our system’s generated answers are not actually grounded in the documents that it retrieves; (2) ELI5 contains significant train / validation overlap, as at least 81% of ELI5 validation questions occur in paraphrased form in the training set; (3) ROUGE-L is not an informative metric of generated answer quality and can be easily gamed; and (4) human evaluations used for other text generation tasks are unreliable for LFQA. We offer suggestions to mitigate each of these issues, which we hope will lead to more rigorous LFQA research and meaningful progress in the future.