Discovering Land Cover Web Map Services from the Deep Web with JavaScript Invocation Rules
Discovering Land Cover Web Map Services from the Deep Web with JavaScript Invocation Rules
复制标题
使用 JavaScript 调用规则从深网发现土地覆盖网络地图服务
DOI：
10.3390/ijgi5070105
复制
发表时间：
2016-06
影响因子：
3.4
通讯作者：
武昊
中科院分区：
文献类型：
--
作者：
武昊
自动发现孤立的土地覆盖网络地图服务（LCWMSs）可能有助于共享土地覆盖数据。目前，已经开发了各种基于搜索引擎和基于爬虫的方法来查找分散在整个表面网络中的服务。事实上，随着地理空间web应用的普及，相当数量的LCWMSs隐藏在JavaScript代码中，属于深网。然而，从JavaScript代码中发现lcwms仍然是一个开放的挑战。本文旨在通过提出一个专注的深度网络爬虫来解决这一挑战，该爬虫可以从深度网络JavaScript代码和表层网络中找到更多的LCWMSs。首先，将一组JavaScript链接的名称抽象为初始判断。通过名称匹配，这些判断被用来判断获取的网页是否包含可能提示JavaScript代码调用wms的预定义JavaScript链接。其次，根据对JavaScript中WMS的使用和编码的先验知识，将WMS的一些JavaScript调用函数和URL格式总结为JavaScript调用规则。这些调用规则用于标识JavaScript代码，以便通过规则匹配提取候选wms。上述两个操作被合并到传统的聚焦爬行策略中，该策略位于抓取网页和解析网页的任务之间。第三，将服务与一组土地覆盖关键字进行匹配，选择LCWMSs。此外，本文还实现了LCWMSs的搜索引擎，该引擎使用深度网络爬虫对发现的LCWMSs进行检索和集成。在第一个实验中，八个在线地理空间web应用程序作为种子url（统一资源定位器）和爬行作用域；建议的爬虫只处理这八个应用程序中的JavaScript代码。所有隐藏在JavaScript代码中的32个可用WMS都是使用建议的爬虫找到的，而通过基于聚焦爬虫的方法没有发现一个WMS。该结果表明，所提出的爬虫具有发现隐藏在JavaScript代码中的wms的能力。第二个实验使用每天更新的4842个种子url。爬虫共发现17,874个可用的WMSs，其中11,901个是LCWMSs。与使用以前的方法相比，我们的方法发现了更多的服务。这表明所提出的爬虫在从表面web和JavaScript代码中发现LCWMSs方面具有很大的优势。此外，一个简单的案例研究表明，所设计的LCWMS搜索引擎是实现全球制图和监测目的的土地覆盖信息集成的重要一步。
Automatic discovery of isolated land cover web map services (LCWMSs) can potentially help in sharing land cover data. Currently, various search engine-based and crawler-based approaches have been developed for finding services dispersed throughout the surface web. In fact, with the prevalence of geospatial web applications, a considerable number of LCWMSs are hidden in JavaScript code, which belongs to the deep web. However, discovering LCWMSs from JavaScript code remains an open challenge. This paper aims to solve this challenge by proposing a focused deep web crawler for finding more LCWMSs from deep web JavaScript code and the surface web. First, the names of a group of JavaScript links are abstracted as initial judgements. Through name matching, these judgements are utilized to judge whether or not the fetched webpages contain predefined JavaScript links that may prompt JavaScript code to invoke WMSs. Secondly, some JavaScript invocation functions and URL formats for WMS are summarized as JavaScript invocation rules from prior knowledge of how WMSs are employed and coded in JavaScript. These invocation rules are used to identify the JavaScript code for extracting candidate WMSs through rule matching. The above two operations are incorporated into a traditional focused crawling strategy situated between the tasks of fetching webpages and parsing webpages. Thirdly, LCWMSs are selected by matching services with a set of land cover keywords. Moreover, a search engine for LCWMSs is implemented that uses the focused deep web crawler to retrieve and integrate the LCWMSs it discovers. In the first experiment, eight online geospatial web applications serve as seed URLs (Uniform Resource Locators) and crawling scopes; the proposed crawler addresses only the JavaScript code in these eight applications. All 32 available WMSs hidden in JavaScript code were found using the proposed crawler, while not one WMS was discovered through the focused crawler-based approach. This result shows that the proposed crawler has the ability to discover WMSs hidden in JavaScript code. The second experiment uses 4842 seed URLs updated daily. The crawler found a total of 17,874 available WMSs, of which 11,901 were LCWMSs. Our approach discovered a greater number of services than those found using previous approaches. It indicates that the proposed crawler has a large advantage in discovering LCWMSs from the surface web and from JavaScript code. Furthermore, a simple case study demonstrates that the designed LCWMS search engine represents an important step towards realizing land cover information integration for global mapping and monitoring purposes.
登录
查看更多内容
影响因子：
2.7
作者：
G. Fenoy;N. Bozon;V. Raghavan
通讯作者：
G. Fenoy;N. Bozon;V. Raghavan
DOI：
10.1109/jstars.2013.2260727
发表时间：
2014
影响因子：
5.5
作者：
T. Stepinski;P. Netzel;J. Jasiewicz
通讯作者：
T. Stepinski;P. Netzel;J. Jasiewicz
DOI：
10.5194/isprsannals-ii-3-w5-167-2015
发表时间：
2015-08
期刊：
ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences
影响因子：
--
作者：
L. See;C. Perger;M. Hofer;J. Weichselbaum;C. Dresel;S. Fritz
通讯作者：
L. See;C. Perger;M. Hofer;J. Weichselbaum;C. Dresel;S. Fritz
DOI：
10.1016/j.envsoft.2012.11.005
发表时间：
2013-03
期刊：
Environ. Model. Softw.
影响因子：
--
作者：
G. Dubois;Michael Schulz;J. Skøien;L. Bastin;S. Peedell
通讯作者：
G. Dubois;Michael Schulz;J. Skøien;L. Bastin;S. Peedell
影响因子：
3.4
作者：
Bartholomé, E;Belward, AS
通讯作者：
Belward, AS