A Reproducible Benchmark for P2P Retrieval
A Reproducible Benchmark for P2P Retrieval
复制标题
P2P 检索的可重复基准
DOI：
--
复制
发表时间：
2006
期刊：
影响因子：
--
通讯作者：
G. Weikum
中科院分区：
文献类型：
--
作者：
Thomas Neumann;Matthias Bender;S. Michel;G. Weikum
随着信息检索在分布式系统中的日益普及，特别是P2P网络搜索，大量的协议和原型已经在文献中介绍。然而，几乎每一篇论文都考虑了不同的实验评估基准，使得它们的相互比较和性能改进的量化成为一项不可能的任务。我们提出了一个标准化的，通用的基准P2P IR系统，最终使这成为可能。我们首先提出了一个详细的需求分析，这样一个标准化的基准框架，允许可重复的和可比的实验设置，而不牺牲灵活性，以适应dierent系统模型。我们进一步建议维基百科作为一个公开的和通用的文档语料库，最后介绍了一个简单但灵活的聚类策略，分配维基百科的文章作为文档的任意数量的同行。在提出了一个标准化的，现实世界的查询集作为基准工作量，我们审查的指标来评估基准结果，并提出了一个例子基准运行我们fullyimplemented P2P Web搜索原型MINERVA。
With the growing popularity of information retrieval (IR) in distributed systems and in particular P2P Web search, a huge number of protocols and prototypes have been introduced in the literature. However, nearly every paper considers a dierent benchmark for its experimental evaluation, rendering their mutual comparison and the quantification of performance improvements an impossible task. We present a standardized, general purpose benchmark for P2P IR systems that finally makes this possible. We start by presenting a detailed requirement analysis for such a standardized benchmark framework that allows for reproducible and comparable experimental setups without sacrificing flexibility to suit dierent system models. We further suggest Wikipedia as a publicly-available and all-purpose document corpus and finally introduce a simple but yet flexible clustering strategy that assigns the Wikipedia articles as documents to an arbitrary number of peers. After proposing a standardized, real-world query set as the benchmark workload, we review the metrics to evaluate the benchmark results and present an example benchmark run for our fullyimplemented P2P Web search prototype MINERVA.