Large-Scale Web Research Testbed

这个项目建立了一个非常大的当前和历史Web内容存储库，支持两组研究工作：Web内容的管理和内容的分析和挖掘。目前的设施在检查万维网（WWW）的许多方面发挥了重要作用。这些方面包括为理解、最佳利用和改进Web而进行的实验。该设施使研究人员能够尝试各种假设和技术来索引和建模WWW信息。系统的高度可配置的爬虫收集大量的Web页面，将它们存储在本地以测试新的算法，例如对集合进行排序、过滤或Web链接映射。目前的WebBase功能不足；例如，爬行速度受到CPU性能的限制（检索到的页面在存储之前会被压缩），通常还会受到虚拟内存空间的限制。消除这两个瓶颈将能够维持更高的Web采样率并覆盖更大的Web区域。一个升级后的测试平台，通过扩大现有系统（称为WebBase）的当前硬件设施的大小和处理速度而开发，将用于研究和评估不同的Web爬行、存档刷新、数据压缩、存储和索引技术。此外，该项目还研究了与数据提取、语义搜索、搜索非文本对象、访问控制、跨时间分析以及挖掘模式或实体之间的关系相关的问题。需要解决的问题包括：如何收集不断增长的Web数据，并使其保持最新；为这些数据提供改进的搜索功能，更好地利用数据和用户请求的语义；有效地处理大容量实时数据流；组织捕获Web“历史”的Web存档；处理新类型的源（例如，隐藏的Web或聊天室）和新类型的数据（例如，图像）。此外，新的WebBase设施将通过提供一个测试平台来支持不同大学的教学，学生可以在这里开发新的搜索、索引和用户表示想法。WebBase汇集了数据挖掘、安全、自然语言处理和数据库系统等领域的教师；因此，这些区域相互增强。因此，基础设施将支持：关键领域的实验研究：Web信息的管理和探索；机构的研究人员没有足够的设备进行大规模的网络抓取；信息检索与数据挖掘课程的教学。

This project, establishing a very large repository of current and historical Web content, supports two group research efforts: Management of the Web content and Analysis and mining of the content.The current facilities have been instrumental in examining many aspects of the World Wide Web (WWW). These aspects include experimentation toward understanding, optimally utilizing, and improving the Web. The facility has enabled researchers to try out various hypothesis and techniques for indexing and modeling WWW information. The system's highly configurable crawlers collect a large number of Web pages, storing them locally for testing novel algorithms, such as ranking, filtering, or Web linkage mapping on the collection. The current WebBase is underpowered; for example, the crawling speeds are limited by CPU performance (retrieved pages are compressed before being stored), and often by virtual memory space. Removing these two bottlenecks will enable sustaining a higher Web sample rate and covering larger areas of the Web. An upgraded testbed, developed by scaling up in size and processing speed of the current hardware facilities of an existing system called WebBase, will be used to study and evaluate different Web crawling, archive refreshing, data compression, and storage and indexing techniques. Moreover, the project investigates problems related to data extraction, semantic search, searching for non-text objects, access control, cross-temporal analysis, and mining patterns or relationships between entities. Problems to be addressed include: How to Collect ever-growing amount of Web data, and keep it up to date, Provide improved search capabilities over such data, better exploiting the semantics of data and user requests, Efficiently process high-volume real-time data streams, Organize a Web archive that captures the "history" of the Web, and Deal with new types of sources (e.g., the hidden web or chat rooms) and new types of data (e.g., images).In addition, the new WebBase facility will support teaching at various universities by providing a testbed where the students can develop new searching, indexing, and user presentation ideas. WebBase draws together faculty in the areas of data mining, security, natural language processing, and database systems; consequently, the areas enhance each other. Thus, the infrastructure will support: Experimental research in a critical area: management and exploration of Web information; Researchers at institutions that do not have sufficient facilities for large-scale Web crawling; and Teaching of courses on information retrieval and data mining.