Large-Scale Web Research Testbed

大规模网络研究测试平台

• 批准号: 0322975
• 负责人: Hector Garcia-Molina
• 金额: $ 43.98万
• 依托单位: Stanford University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2003
• 资助国家: 美国
• 起止时间: 2003-09-15 至 2006-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0322975&HistoricalAwards=false
• 关键词: Large; Scale; Web; Research; Testbed

This project, establishing a very large repository of current and historical Web content, supports two group research efforts: Management of the Web content and Analysis and mining of the content.The current facilities have been instrumental in examining many aspects of the World Wide Web (WWW). These aspects include experimentation toward understanding, optimally utilizing, and improving the Web. The facility has enabled researchers to try out various hypothesis and techniques for indexing and modeling WWW information. The system's highly configurable crawlers collect a large number of Web pages, storing them locally for testing novel algorithms, such as ranking, filtering, or Web linkage mapping on the collection. The current WebBase is underpowered; for example, the crawling speeds are limited by CPU performance (retrieved pages are compressed before being stored), and often by virtual memory space. Removing these two bottlenecks will enable sustaining a higher Web sample rate and covering larger areas of the Web. An upgraded testbed, developed by scaling up in size and processing speed of the current hardware facilities of an existing system called WebBase, will be used to study and evaluate different Web crawling, archive refreshing, data compression, and storage and indexing techniques. Moreover, the project investigates problems related to data extraction, semantic search, searching for non-text objects, access control, cross-temporal analysis, and mining patterns or relationships between entities. Problems to be addressed include: How to Collect ever-growing amount of Web data, and keep it up to date, Provide improved search capabilities over such data, better exploiting the semantics of data and user requests, Efficiently process high-volume real-time data streams, Organize a Web archive that captures the "history" of the Web, and Deal with new types of sources (e.g., the hidden web or chat rooms) and new types of data (e.g., images).In addition, the new WebBase facility will support teaching at various universities by providing a testbed where the students can develop new searching, indexing, and user presentation ideas. WebBase draws together faculty in the areas of data mining, security, natural language processing, and database systems; consequently, the areas enhance each other. Thus, the infrastructure will support: Experimental research in a critical area: management and exploration of Web information; Researchers at institutions that do not have sufficient facilities for large-scale Web crawling; and Teaching of courses on information retrieval and data mining.

该项目建立了一个非常大的当前和历史 Web 内容存储库，支持两个小组的研究工作：Web 内容管理以及内容分析和挖掘。当前的设施在检查万维网 (WWW) 的许多方面发挥了重要作用。这些方面包括理解、优化利用和改进网络的实验。该设施使研究人员能够尝试各种假设和技术来对 WWW 信息进行索引和建模。该系统的高度可配置的爬虫收集大量网页，将它们存储在本地以测试新颖的算法，例如集合上的排名、过滤或Web链接映射。当前的 WebBase 功能不足；例如，抓取速度受到CPU性能的限制（检索到的页面在存储之前被压缩），并且通常受到虚拟内存空间的限制。消除这两个瓶颈将能够维持更高的网络采样率并覆盖更大的网络区域。通过扩大现有系统（称为 WebBase）的当前硬件设施的规模和处理速度而开发的升级测试平台将用于研究和评估不同的网络爬行、存档刷新、数据压缩以及存储和索引技术。此外，该项目还研究与数据提取、语义搜索、搜索非文本对象、访问控制、跨时空分析以及挖掘模式或实体之间的关系相关的问题。需要解决的问题包括：如何收集不断增长的网络数据量并使其保持最新；提供对此类数据改进的搜索功能，更好地利用数据和用户请求的语义；有效处理大量实时数据流；组织捕获网络“历史”的网络档案；以及处理新类型的源（例如，隐藏的网络或聊天室）和新类型的数据（例如，图像）。 此外，新的 WebBase 设施将通过提供一个测试平台来支持各大学的教学，学生可以在其中开发新的搜索、索引和用户演示想法。 WebBase 汇集了数据挖掘、安全、自然语言处理和数据库系统领域的教师；因此，这些领域相互促进。因此，该基础设施将支持： 关键领域的实验研究：网络信息的管理和探索；没有足够设施进行大规模网络爬行的机构的研究人员；信息检索和数据挖掘课程的教学。