SBIR Phase I: Real-Time Data Analytics Over The Deep Web

这个小企业创新研究（SBIR）第一阶段项目解决了通过深网存储库进行实时数据分析的问题。深网的主要部分由隐藏在限制性网络搜索界面后面的在线数据存储库组成，因此无法被现有的搜索引擎（如Google）有效地抓取。所提出的技术使用基于采样的框架，通过发出少量的搜索请求，通过现有的深网存储库的Web接口，快速生成深网分析的可见的副本。具体的技术目标包括能够“深入”一个小的感兴趣的子区域，并以最小的查询成本下载所需的数据，以及能够从深网存储库自动提取元数据信息。预期的技术成果是算法，该算法在通过深网存储库的搜索接口发出预定数量的请求后发现感兴趣的数据和/或元数据信息。通过认识到该项目使金融、政治、经济、金融和金融等市场部门民主化，以及市场分析，否则需要大量的人力和/或计算资源，例如高薪的主题专家或服务器以及用于web爬行/索引的存储器。而不是产生这样的高成本，所提出的技术为客户提供了一个负担得起的解决方案，用于多个深网数据存储库的实时聚合分析。更广泛地说，各种企业、政府和情报机构的知识工作者都需要通过深网进行实时数据分析。通过该项目增强科学和技术理解，使公众有能力在深网上提出高层次的分析查询，这一前景对整个社会都是诱人和有益的。

This Small Business Innovation Research (SBIR) Phase I project addresses the problem of real-time data analytics over deep web repositories. A major part of the deep web consists of online data repositories that are hidden behind restrictive web search interfaces and therefore cannot be effectively crawled by existing search engines such as Google. The proposed technology uses a sampling-based framework to quickly generate visible depictions of deep web analytics by issuing a small number of search requests though the existing web interfaces of deep web repositories. The specific technical objectives include the ability to 'drill into' a small subarea of interest and download the desired data with minimal query cost, as well as the ability to extract metadata information automatically from a deep web repository. The anticipated technical results are algorithms that discover the data of interest and/or metadata information after issuing a predetermined number of requests through the search interfaces of deep web repositories.The broader impact/commercial potential of this project is understood by recognizing that it democratizes the market sectors of financial, political, and market analysis which would otherwise require heavy human effort and/or computing resources such as highly paid subject matter experts or servers and storage for web crawling/indexing. Instead of incurring such high costs, the proposed technology provides customers an affordable solution for the real-time aggregate analysis of multiple deep web data repositories. More broadly, real-time data analytics over the deep web is needed by knowledge workers in a wide variety of corporations, governments, and intelligence agencies. The prospects of empowering the general public with the ability to pose high-level analytical queries over the deep web, using the enhanced scientific and technological understanding achieved through this project, are tantalizing and beneficial to the entire society at large.