Toward Reproducible Baselines: The Open-Source IR Reproducibility Challenge
Toward Reproducible Baselines: The Open-Source IR Reproducibility Challenge
复制标题
迈向可重现的基线：开源红外重现性挑战
DOI：
--
复制
发表时间：
2016
期刊：
影响因子：
--
通讯作者：
S. Vigna
中科院分区：
文献类型：
--
作者：
Jimmy J. Lin;Matt Crane;A. Trotman;Jamie Callan;Ishan Chattopadhyaya;John Foley;Grant Ingersoll;Craig Macdonald;S. Vigna
Open-Source IR Reproducibility Challenge将开源搜索引擎的开发人员聚集在一起，在Amazon EC2上的公共环境中提供其系统的可复制基线。该产品是一个存储库，其中包含生成竞争性特定检索基线所需的所有代码，因此使用单个脚本，任何拥有集合副本的人都可以复制提交的运行。我们的愿景是，这些结果将作为广泛访问的比较点，在未来的IR研究。这个项目代表了一个持续的努力，但我们描述的第一阶段的挑战，是在SIGIR 2015研讨会的一部分组织。到目前为止，我们取得了一定的成功，通过七个开源搜索引擎实现了我们在Gov 2收集方面的主要目标。在本文中，我们描述了我们的方法，分享实验结果，并讨论了经验教训以及下一步。
The Open-Source IR Reproducibility Challenge brought together developers of open-source search engines to provide reproducible baselines of their systems in a common environment on Amazon EC2. The product is a repository that contains all code necessary to generate competitive ad hoc retrieval baselines, such that with a single script, anyone with a copy of the collection can reproduce the submitted runs. Our vision is that these results would serve as widely accessible points of comparison in future IR research. This project represents an ongoing effort, but we describe the first phase of the challenge that was organized as part of a workshop at SIGIR 2015. We have succeeded modestly so far, achieving our main goals on the Gov2 collection with seven open-source search engines. In this paper, we describe our methodology, share experimental results, and discuss lessons learned as well as next steps.