SGER: Multi-Tier Indexing for Web Search Engines

这个项目对先前的联邦搜索工作进行了调整，以创建一种更具选择性的方法来搜索web索引，我们称之为主题分区索引。主题分区索引的每个子集（分片）覆盖特定的内容区域，因此只有分片覆盖查询？需要搜索5个主题区域。我们的研究是开发有效地将文档分配到分片的方法。有监督和无监督技术用于将查询匹配到分片。结果是一种选择性搜索，它提供了与更详尽的搜索相似的准确性，但需要的工作量少了一个数量级，从而产生了显著的计算和财务节省。该项目使用b谷歌/IBM集群来抓取网络，并执行必要的数据清理和预处理，以开发一个包含5亿到10亿个文档的网络数据集来支持研究。正在作出额外的努力，以编制一个对广泛的研究目的有用的语料库。项目目标是与b谷歌/IBM集群上的其他研究人员共享数据集，并最终与更广泛的研究社区共享数据集。该项目将产生三种类型的广泛影响。大型网络搜索公司的数据中心价格昂贵，是电力的主要消费者，因此降低其成本具有显著的财务和环境效益。较低的计算成本使学术研究人员能够对网络搜索公司认为可信的数据集进行研究，从而增加学术研究的影响。最后，像我们这样的研究数据集通常具有很长的寿命，并被世界各地的科学家用于各种研究项目。

This project is adapting prior work on federated search to create a more selective approach to searching web indexes that we call topic-partitioned indexing. Each subset (shard) of a topic-partitioned index covers specific content areas, so that only shards covering the query?s topic area(s) need to be searched. Our research is developing methods to efficiently assign documents to shards. Supervised and unsupervised techniques are used to match queries to shards. The result is a selective search that delivers similar accuracy as more exhaustive searches, but requires an order of magnitude less effort, thus yielding significant computational and financial savings. The project is using the Google/IBM cluster to crawl the web and perform the data cleansing and pre-processing necessary to develop a web dataset of 500 million to 1 billion documents to support the research. Additional effort is being devoted to producing a corpus that is useful for a broad range of research purposes. A project goal is to share the dataset with other researchers on the Google/IBM cluster, and eventually with a broader research community.The project will have three types of broad impact. The data centers of large web search companies are expensive and major consumers of electrical power, thus reducing their costs has significant financial and environmental benefits. Lower computational costs make it practical for academic researchers to conduct research on datasets that web search companies consider credible, thus increasing the impact of academic research. Finally, research datasets such ours typically have long life spans and are used for diverse research projects by scientists around the world.