Geographical partition for distributed web crawling
Geographical partition for distributed web crawling
复制标题
分布式网络爬取的地理分区
DOI：
10.1145/1096985.1096999
复制
发表时间：
2005
期刊：
影响因子：
--
通讯作者：
J. Rufino
中科院分区：
文献类型：
--
作者：
J. Exposto;J. Macedo;A. Pina;A. Alves;J. Rufino
本文通过对Web的地理分区来评估可扩展的分布式爬行。该方法是基于多个分布式爬虫的存在，每个爬虫负责属于一个或多个先前识别的地理区域的页面。这项工作认为，分布式爬虫的网页访问的分配是基于页面内容的地理范围。对于一个页面到一个分区的初始分配，我们使用一个简单的启发式方法，该方法将一个页面标记在托管Web服务器地理位置的同一范围内。在下载过程中，如果对页面内容的分析建议了不同的地理范围，则将页面转发到位置良好的Web服务器。2005年期间提取的葡萄牙语网页样本用于评估：a）页面下载通信时间和B）服务器之间页面交换的开销。评估结果允许比较我们的方法，传统的哈希分区策略。
This paper evaluates scalable distributed crawling by means of the geographical partition of the Web. The approach is based on the existence of multiple distributed crawlers each one responsible for the pages belonging to one or more previously identified geographical zones. The work considers a distributed crawler where the assignment of pages to visit is based on page content geographical scope. For the initial assignment of a page to a partition we use a simple heuristic that marks a page within the same scope of the hosting web server geographical location. During download, if the analyze of a page contents recommends a different geographical scope, the page is forwarded to the well-located web server.A sample of the Portuguese Web pages, extracted during the year 2005, was used to evaluate: a) page download communication times and the b) overhead of pages exchange among servers. Evaluation results permit to compare our approach to conventional hash partitioning strategies.