Humans Optional? Automatic Large-Scale Test Collections for Entity, Passage, and Entity-Passage Retrieval
Humans Optional? Automatic Large-Scale Test Collections for Entity, Passage, and Entity-Passage Retrieval
复制标题
DOI：
10.1007/s13222-020-00334-y
复制
发表时间：
2020-03
期刊：
影响因子：
--
通讯作者：
Laura Dietz;Jeffrey Dalton
中科院分区：
文献类型：
--
作者：
Laura Dietz;Jeffrey Dalton
手动创建测试集合是一个耗时、精力和成本密集的过程。本文描述了一种用于派生大规模测试集合的全自动替代方案，无需人工评估。实证实验证实，自动测试收集和手动评估在最佳性能系统上是一致的。该集合包括文本段落和知识库实体的相关性判断。由于具有实体和文本段落相关数据的测试集合很少，因此这种方法为训练和评估临时段落检索、实体检索和实体感知文本检索方法提供了一种经济有效的方法。
Manually creating test collections is a time-, effort-, and cost-intensive process. This paper describes a fully automatic alternative for deriving large-scale test collections, where no human assessments are needed. The empirical experiments confirm that automatic test collection and manual assessments agree on the best performing systems. The collection includes relevance judgments for both text passages and knowledge base entities. Since test collections with relevance data for both entity and text passages are rare, this approach provides a cost-efficient way for training and evaluating ad hoc passage retrieval, entity retrieval, and entity-aware text retrieval methods.