Lexical profiling of existing web directories to support fine-grained topic-focused web crawling
Lexical profiling of existing web directories to support fine-grained topic-focused web crawling
复制标题
对现有 Web 目录进行词法分析，以支持以主题为中心的细粒度 Web 爬行
DOI：
10.14236/ewic/irsg2008.4
复制
发表时间：
2008
期刊：
影响因子：
3.7
通讯作者：
G. Nenadic
中科院分区：
文献类型：
--
作者：
M. Greenwood;G. Nenadic
以主题为中心的Web爬行旨在可靠和高效地利用互联网的潜力，生成Web内页面的特定主题索引。以前的工作侧重于提供适当的一般主题描述，以生成较大的一般索引。在本文中，我们提出了一种方法，该方法利用现有Web目录中包含层次结构的语料库进行词汇剖析，在较小的训练样本上指定更细粒度的主题，同时利用相关主题中看似冗余的信息来提高页面收集过程的效率。我们还建议了一个链接评分公式，该公式结合了内容、上下文和页面词汇与给定主题的相似性，以确定爬行链接的优先级。开放目录项目的初步实验表明，优先爬行器提供的页面明显多于广度优先爬行器。此外，相关页面数量的增长速度也要高得多。让爬虫靠近目标主题，通过跟踪最有可能链接到目标页面的链接，可以减少“非生产性”时间。
Topic-focused Web crawling aims to harness the potential of the Internet reliably and efficiently, producing topic specific indexes of pages within the Web. Previous work has focused on supplying suitably general descriptions of topics to generate large general indexes. In this paper we propose a method that uses lexical profiling of a corpus that consists of hierarchical structures in existing Web Directories to specify finer-grained topics on smaller training examples, while using the seemingly redundant information in related topics to make the process of gathering pages more efficient. We also suggest a link scoring formula that combines content, context and page lexical similarities to a given topic to prioritise the links for crawling. The initial experiments with the Open Directory Project show that the prioritised crawl provides significantly more pages than the breadth-first crawler. Also, the rate at which the number of relevant pages increases is much higher. Keeping the crawler close to the target subject allows "unproductive" periods to be reduced, by following links most likely to link to target pages.