Large-Scale Web Research Testbed

这个项目建立了一个非常大的当前和历史Web内容储存库，支持两个小组的研究工作：Web内容的管理和内容的分析和挖掘。目前的设施在检查万维网(WWW)的许多方面起到了重要作用。这些方面包括了解、最佳利用和改进Web的实验。该设施使研究人员能够尝试各种假设和技术，为WWW信息建立索引和建模。该系统高度可配置的爬虫程序收集大量Web页面，将它们存储在本地，用于测试新的算法，如对集合进行排名、过滤或Web链接映射。当前的WebBase功能不足；例如，爬行速度受到CPU性能的限制(检索到的页面在存储之前会被压缩)，而且通常还受到虚拟内存空间的限制。消除这两个瓶颈将能够保持更高的Web采样率并覆盖更大的Web区域。通过扩大现有系统WebBase的现有硬件设施的规模和处理速度而开发的升级试验床将用于研究和评估不同的网络爬行、档案刷新、数据压缩以及存储和索引技术。此外，该项目还调查与数据提取、语义搜索、搜索非文本对象、访问控制、跨时间分析以及挖掘模式或实体之间的关系有关的问题。要解决的问题包括：如何收集不断增长的Web数据，并使其保持最新；提供改进的对这些数据的搜索能力；更好地利用数据和用户请求的语义；高效地处理大量实时数据流；组织捕获Web“历史”的Web档案；处理新类型的源(例如，隐藏的Web或聊天室)和新类型的数据(例如，图像)。此外，新的WebBase设施将通过提供一个试验台来支持不同大学的教学，学生可以在那里开发新的搜索、索引、和用户演示想法。WebBase汇集了数据挖掘、安全、自然语言处理和数据库系统等领域的教师；因此，这些领域相互促进。因此，该基础设施将支持以下关键领域的实验研究：网络信息的管理和探索；没有足够设施进行大规模网络爬行的机构的研究人员；信息检索和数据挖掘课程的教学。

This project, establishing a very large repository of current and historical Web content, supports two group research efforts: Management of the Web content and Analysis and mining of the content.The current facilities have been instrumental in examining many aspects of the World Wide Web (WWW). These aspects include experimentation toward understanding, optimally utilizing, and improving the Web. The facility has enabled researchers to try out various hypothesis and techniques for indexing and modeling WWW information. The system's highly configurable crawlers collect a large number of Web pages, storing them locally for testing novel algorithms, such as ranking, filtering, or Web linkage mapping on the collection. The current WebBase is underpowered; for example, the crawling speeds are limited by CPU performance (retrieved pages are compressed before being stored), and often by virtual memory space. Removing these two bottlenecks will enable sustaining a higher Web sample rate and covering larger areas of the Web. An upgraded testbed, developed by scaling up in size and processing speed of the current hardware facilities of an existing system called WebBase, will be used to study and evaluate different Web crawling, archive refreshing, data compression, and storage and indexing techniques. Moreover, the project investigates problems related to data extraction, semantic search, searching for non-text objects, access control, cross-temporal analysis, and mining patterns or relationships between entities. Problems to be addressed include: How to Collect ever-growing amount of Web data, and keep it up to date, Provide improved search capabilities over such data, better exploiting the semantics of data and user requests, Efficiently process high-volume real-time data streams, Organize a Web archive that captures the "history" of the Web, and Deal with new types of sources (e.g., the hidden web or chat rooms) and new types of data (e.g., images).In addition, the new WebBase facility will support teaching at various universities by providing a testbed where the students can develop new searching, indexing, and user presentation ideas. WebBase draws together faculty in the areas of data mining, security, natural language processing, and database systems; consequently, the areas enhance each other. Thus, the infrastructure will support: Experimental research in a critical area: management and exploration of Web information; Researchers at institutions that do not have sufficient facilities for large-scale Web crawling; and Teaching of courses on information retrieval and data mining.