CI-ADDO-EN: Collaborative Proposal: Supporting Web-Scale Experimentation using the Lemur Toolkit

点击翻译按钮获取中文摘要

This project maintains and enhances existing community softwareinfrastructure, and creates new community data infrastructure toenable the information retrieval research community and relatedresearch communities to conduct research on a "web scale",meaning datasets of a billion or more web pages together withlarge query logs. The software infrastructure is based on theLemur Toolkit and the associated Indri search engine, which areused by many information retrieval researchers due to the supportfor multiple retrieval models, multiple forms of evidence, and apowerful probabilistic query language. The enhancements to Lemurinclude support for the popular MapReduce style of distributedprocessing and other efficiency improvements to make it practicalto do research on large web datasets 'out of the box' in commoncomputer hardware environments.The new data infrastructure consists of maintenance anddistribution of a newly created billion-page dataset, another newweb dataset, and large, anonymized search logs that match thedatasets. The combination of large datasets and correspondinglarge search logs enable a broad community to conduct researchwith more realistic data resources than were availablepreviously. This research will lead to further advances in theunderstanding of the underlying issues for large-scale,personalized search, which will be an important part of the nextgeneration of search engines.For further information, see the project web site at the URL:http://www.lemurproject.org.