CI-ADDO-EN: Collaborative Proposal: Supporting Web-Scale Experimentation using the Lemur Toolkit

该项目维护和增强现有的社区软件基础设施，并创建新的社区数据基础设施，使信息检索研究社区和相关研究社区能够在“网络规模”上进行研究，这意味着十亿或更多网页的数据集以及大型查询日志。软件基础设施基于Lemur Toolkit和相关的Indri搜索引擎，由于支持多种检索模型，多种形式的证据和强大的概率查询语言，许多信息检索研究人员都在使用它们。Lemur的增强功能包括支持流行的MapReduce分布式处理风格和其他效率改进，使其能够在普通计算机硬件环境中对大型Web数据集进行“开箱即用”的研究。新的数据基础设施包括维护和分发一个新创建的十亿页数据集，另一个新的Web数据集，以及与数据集匹配的大型匿名搜索日志。大型数据集和相应的大型搜索日志的组合使广泛的社区能够使用比以前更真实的数据资源进行研究。这项研究将导致对大规模个性化搜索的基本问题的进一步理解，这将是下一代搜索引擎的重要组成部分。http://www.lemurproject.org

This project maintains and enhances existing community softwareinfrastructure, and creates new community data infrastructure toenable the information retrieval research community and relatedresearch communities to conduct research on a "web scale",meaning datasets of a billion or more web pages together withlarge query logs. The software infrastructure is based on theLemur Toolkit and the associated Indri search engine, which areused by many information retrieval researchers due to the supportfor multiple retrieval models, multiple forms of evidence, and apowerful probabilistic query language. The enhancements to Lemurinclude support for the popular MapReduce style of distributedprocessing and other efficiency improvements to make it practicalto do research on large web datasets 'out of the box' in commoncomputer hardware environments.The new data infrastructure consists of maintenance anddistribution of a newly created billion-page dataset, another newweb dataset, and large, anonymized search logs that match thedatasets. The combination of large datasets and correspondinglarge search logs enable a broad community to conduct researchwith more realistic data resources than were availablepreviously. This research will lead to further advances in theunderstanding of the underlying issues for large-scale,personalized search, which will be an important part of the nextgeneration of search engines.For further information, see the project web site at the URL:http://www.lemurproject.org.