SBIR Phase I: Real-Time Data Analytics Over The Deep Web

这个小型企业创新研究(SBIR)第一阶段项目解决了在深度Web存储库上进行实时数据分析的问题。深网的一个主要部分是隐藏在受限的网络搜索界面后面的在线数据仓库，因此无法被谷歌等现有搜索引擎有效地抓取。提议的技术使用基于抽样的框架，通过通过现有的深度网络储存库的网络接口发出少量搜索请求来快速生成对深度网络分析的可视描述。具体的技术目标包括能够“钻取”感兴趣的一小块区域并以最小的查询成本下载所需数据，以及能够从深层网络储存库自动提取元数据信息。预期的技术成果是在通过深层网络仓库的搜索接口发出预定数量的请求后发现感兴趣的数据和/或元数据信息的算法。该项目的更广泛的影响/商业潜力是通过认识到它使金融、政治和市场分析的市场部门民主化而理解的，否则这些市场部门将需要大量的人力和/或计算资源，例如高薪主题专家或用于网络爬行/索引的服务器和存储。该技术不会产生如此高的成本，而是为客户提供了一种负担得起的解决方案，用于对多个深层网络数据存储库进行实时聚合分析。更广泛地说，各种公司、政府和情报机构的知识工作者都需要对深层网络进行实时数据分析。利用通过该项目实现的增进对科学和技术的理解，使普通公众能够在深层网络上提出高级分析性查询的前景是诱人的，对整个社会都是有益的。

This Small Business Innovation Research (SBIR) Phase I project addresses the problem of real-time data analytics over deep web repositories. A major part of the deep web consists of online data repositories that are hidden behind restrictive web search interfaces and therefore cannot be effectively crawled by existing search engines such as Google. The proposed technology uses a sampling-based framework to quickly generate visible depictions of deep web analytics by issuing a small number of search requests though the existing web interfaces of deep web repositories. The specific technical objectives include the ability to 'drill into' a small subarea of interest and download the desired data with minimal query cost, as well as the ability to extract metadata information automatically from a deep web repository. The anticipated technical results are algorithms that discover the data of interest and/or metadata information after issuing a predetermined number of requests through the search interfaces of deep web repositories.The broader impact/commercial potential of this project is understood by recognizing that it democratizes the market sectors of financial, political, and market analysis which would otherwise require heavy human effort and/or computing resources such as highly paid subject matter experts or servers and storage for web crawling/indexing. Instead of incurring such high costs, the proposed technology provides customers an affordable solution for the real-time aggregate analysis of multiple deep web data repositories. More broadly, real-time data analytics over the deep web is needed by knowledge workers in a wide variety of corporations, governments, and intelligence agencies. The prospects of empowering the general public with the ability to pose high-level analytical queries over the deep web, using the enhanced scientific and technological understanding achieved through this project, are tantalizing and beneficial to the entire society at large.