Deep Web adaptive crawling based on minimum executable pattern
Deep Web adaptive crawling based on minimum executable pattern
复制标题
DOI：
10.1007/s10844-010-0124-5
复制
发表时间：
2011-04
影响因子：
3.4
通讯作者：
Jun Liu;Lu Jiang;Zhaohui Wu;Q. Zheng
中科院分区：
文献类型：
--
作者：
Jun Liu;Lu Jiang;Zhaohui Wu;Q. Zheng
Deep Web Crawling的关键是将有效的输入值提交到查询表单并有效地检索Deep Web内容。在文献中，相关的工作只集中在通用的文本框或整个查询表单，导致“数据孤岛”或查询提交的有效性较差的问题。提出了最小可执行模式（MEP）的概念，即查询表单中能够进行成功查询的元素的最小组合，并给出了MEP生成方法和基于MEP的Deep Web自适应爬取方法。首先对查询表单进行解析并划分成MEP集合，然后通过在MEP集合中选择一个MEP和该MEP的关键字向量来生成局部最优查询。此外，爬虫可以决定其终止，以平衡高覆盖率的内容和资源消耗之间的权衡。MEP的采用有望提高查询提交的有效性，并且多个MEP的自适应选择对于克服“数据孤岛”问题具有良好的效果。我们提出了一组实验来验证所提出的方法的有效性。实验结果表明，我们的方法优于最先进的方法在查询能力和适用性方面，平均而言，它实现了良好的覆盖率发出只有几百个查询。
The key to Deep Web Crawling is to submit valid input values to a query form and retrieve Deep Web content efficiently. In the literature, related work focus only on generic text boxes or entire query forms, causing the problem of “data islands” or inferior validity of query submission. This paper proposes the concept of Minimum Executable Pattern (MEP), a minimal combination of elements in a query form that can conduct a successful query, and then presents a MEPGeneration method and a MEP-based Deep Web adaptive crawling method. The query form is parsed and partitioned into MEP set, and then local-optimal queries are generated by choosing a MEP in the MEP set and a keyword vector of the MEP. Furthermore, the crawler can make a decision on its termination to balance the trade-off between high coverage of the content and resource consumption. The adoption of MEP is expected to improve the validity of query submission, and adaptive selection of multiple MEPs shows good effect for overcoming the problem of “data islands”. We present a set of experiments to validate the effectiveness of the proposed method. Experimental results show that our method outperforms the state of art methods in terms of query capability and applicability, and on average, it achieves good coverage by issuing only a few hundred queries.