SGER: Multi-Tier Indexing for Web Search Engines

这个项目正在改编之前关于联邦搜索的工作，以创建一种更具选择性的方法来搜索Web索引，我们称之为主题分区索引。主题分区索引的每个子集（分片）都覆盖特定的内容区域，因此只有分片覆盖查询？的主题区域需要搜索。我们的研究正在开发有效地将文档分配到分片的方法。监督和无监督技术用于将查询匹配到分片。其结果是一个选择性的搜索，提供类似的准确性，更详尽的搜索，但需要一个数量级更少的努力，从而产生显着的计算和财务节省。该项目使用Google/IBM集群来抓取Web，并执行必要的数据清理和预处理，以开发包含5亿至10亿个文档的Web数据集来支持研究。目前正在进一步努力制作一个对广泛研究有用的语料库。该项目的目标是与Google/IBM集群上的其他研究人员共享数据集，并最终与更广泛的研究社区共享数据集。大型网络搜索公司的数据中心是昂贵的，并且是电力的主要消耗者，因此降低其成本具有显著的经济和环境效益。较低的计算成本使学术研究人员能够对网络搜索公司认为可信的数据集进行研究，从而增加学术研究的影响力。最后，像我们这样的研究数据集通常具有很长的生命周期，并被世界各地的科学家用于各种研究项目。

This project is adapting prior work on federated search to create a more selective approach to searching web indexes that we call topic-partitioned indexing. Each subset (shard) of a topic-partitioned index covers specific content areas, so that only shards covering the query?s topic area(s) need to be searched. Our research is developing methods to efficiently assign documents to shards. Supervised and unsupervised techniques are used to match queries to shards. The result is a selective search that delivers similar accuracy as more exhaustive searches, but requires an order of magnitude less effort, thus yielding significant computational and financial savings. The project is using the Google/IBM cluster to crawl the web and perform the data cleansing and pre-processing necessary to develop a web dataset of 500 million to 1 billion documents to support the research. Additional effort is being devoted to producing a corpus that is useful for a broad range of research purposes. A project goal is to share the dataset with other researchers on the Google/IBM cluster, and eventually with a broader research community.The project will have three types of broad impact. The data centers of large web search companies are expensive and major consumers of electrical power, thus reducing their costs has significant financial and environmental benefits. Lower computational costs make it practical for academic researchers to conduct research on datasets that web search companies consider credible, thus increasing the impact of academic research. Finally, research datasets such ours typically have long life spans and are used for diverse research projects by scientists around the world.