SGER: Multi-Tier Indexing for Web Search Engines

SGER：网络搜索引擎的多层索引

• 批准号: 0841275
• 负责人: Jamie Callan
• 金额: $ 20万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2008
• 资助国家: 美国
• 起止时间: 2008-08-01 至 2010-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0841275&HistoricalAwards=false
• 关键词: SGER; Multi; Tier; Indexing; Web

This project is adapting prior work on federated search to create a more selective approach to searching web indexes that we call topic-partitioned indexing. Each subset (shard) of a topic-partitioned index covers specific content areas, so that only shards covering the query?s topic area(s) need to be searched. Our research is developing methods to efficiently assign documents to shards. Supervised and unsupervised techniques are used to match queries to shards. The result is a selective search that delivers similar accuracy as more exhaustive searches, but requires an order of magnitude less effort, thus yielding significant computational and financial savings. The project is using the Google/IBM cluster to crawl the web and perform the data cleansing and pre-processing necessary to develop a web dataset of 500 million to 1 billion documents to support the research. Additional effort is being devoted to producing a corpus that is useful for a broad range of research purposes. A project goal is to share the dataset with other researchers on the Google/IBM cluster, and eventually with a broader research community.The project will have three types of broad impact. The data centers of large web search companies are expensive and major consumers of electrical power, thus reducing their costs has significant financial and environmental benefits. Lower computational costs make it practical for academic researchers to conduct research on datasets that web search companies consider credible, thus increasing the impact of academic research. Finally, research datasets such ours typically have long life spans and are used for diverse research projects by scientists around the world.

该项目正在修改联合搜索方面的先前工作，以创建一种更具选择性的搜索Web索引的方法，我们称之为主题分区索引。主题分区索引的每个子集(分片)覆盖特定的内容区域，因此只需要搜索覆盖查询的分片？S主题区域(S)。我们的研究正在开发有效地将文档分配到分片的方法。使用监督和非监督技术将查询与碎片进行匹配。结果是选择性搜索提供了与更详尽的搜索类似的精确度，但需要的工作量少了一个数量级，从而产生了显著的计算和财务节省。该项目正在使用谷歌/IBM集群来爬行网络，并执行必要的数据清理和预处理，以开发一个包含5亿至10亿份文档的网络数据集，以支持这项研究。目前正在致力于建立一个对广泛研究目的有用的语料库。该项目的目标是与谷歌/IBM集群上的其他研究人员共享数据集，并最终与更广泛的研究社区共享。该项目将产生三种类型的广泛影响。大型网络搜索公司的数据中心价格昂贵，也是电力的主要消耗者，因此降低它们的成本具有显著的财务和环境效益。较低的计算成本使学术研究人员能够对网络搜索公司认为可信的数据集进行研究，从而增加了学术研究的影响。最后，像我们这样的研究数据集通常具有很长的寿命，并被世界各地的科学家用于各种研究项目。