Evaluation of crawling policies for a web-repository crawler
Evaluation of crawling policies for a web-repository crawler
复制标题
网络存储库爬虫的爬行策略评估
DOI：
10.1145/1149941.1149972
复制
发表时间：
2006
期刊：
影响因子：
3.7
通讯作者：
Michael L. Nelson
中科院分区：
文献类型：
--
作者：
F. McCown;Michael L. Nelson
我们开发了一个网络存储库爬虫，用于在备份不可用时重建网站。我们的爬虫检索网络资源从互联网档案馆，谷歌，雅虎和MSN。我们研究了抓取Web存储库的挑战，并讨论了克服这些障碍的策略。我们提出了三个爬行策略，可用于重建网站。我们通过重建24个网站并将结果与网站的实时版本进行比较来评估政策的有效性。最后，我们代表他人重建丢失的网站的经验，并讨论改进我们的网络存储库爬虫的计划。
We have developed a web-repository crawler that is used for reconstructing websites when backups are unavailable. Our crawler retrieves web resources from the Internet Archive, Google, Yahoo and MSN. We examine the challenges of crawling web repositories, and we discuss strategies for overcoming some of these obstacles. We propose three crawling policies which can be used to reconstruct websites. We evaluate the effectiveness of the policies by reconstructing 24 websites and comparing the results with live versions of the websites. We conclude with our experiences reconstructing lost websites on behalf of others and discuss plans for improving our web-repository crawler.