ANTIQUE: A Non-factoid Question Answering Benchmark
ANTIQUE: A Non-factoid Question Answering Benchmark
复制标题
DOI：
10.1007/978-3-030-45442-5_21
复制
发表时间：
2020-03-24
期刊：
影响因子：
--
通讯作者：
Croft WB
中科院分区：
文献类型：
--
作者：
Hashemi H;Aliannejadi M;Zamani H;Croft WB
随着移动的和语音搜索的广泛应用，非事实问题的答案段检索在现代信息检索系统中起着至关重要的作用。尽管这项任务很重要，但社区仍然感到缺乏大规模的非事实问答集，这些问答集具有真实的问题和全面的相关性判断。在本文中，我们开发并发布了一系列来自不同类别的2，626个开放域非事实问题。该数据集名为ANTIQUE，包含34，000个手动相关性注释。这些问题是由社区问答服务中的真实的用户提出的，即，耶！答案每个问题的所有答案的相关性判断是通过众包收集的。为了便于进一步的研究，我们还包括对经典和神经IR模型的数据以及基线结果的简要分析。
Considering the widespread use of mobile and voice search, answer passage retrieval for non-factoid questions plays a critical role in modern information retrieval systems. Despite the importance of the task, the community still feels the significant lack of large-scale non-factoid question answering collections with real questions and comprehensive relevance judgments. In this paper, we develop and release a collection of 2,626 open-domain non-factoid questions from a diverse set of categories. The dataset, called ANTIQUE, contains 34k manual relevance annotations. The questions were asked by real users in a community question answering service, i.e., Yahoo! Answers. Relevance judgments for all the answers to each question were collected through crowdsourcing. To facilitate further research, we also include a brief analysis of the data as well as baseline results on both classical and neural IR models.