STTR Phase I: Real-Time Smart Data Lake

这个小企业技术转移（STTR）第一阶段项目的广泛影响将从根本上加快数据科学的步伐。需要新的解决方案来缓解或绕过现有数据分析技术固有的瓶颈，以释放“大数据”所包含的价值，并为研究、商业和整个社会带来更大的利益。潜在的好处包括改进制成品的质量控制，减少金融交易中的欺诈行为，以及增强消费者服务的定制化。许多常见的软件应用程序，如搜索引擎、在线购物、电子商务、医疗应用程序和社交网络，都由数据分析处理服务提供支持。该项目将允许共享存储服务直接对大量数据集进行预处理，然后允许客户端分析应用程序有效地利用这一功能。该项目将加速分析数据向有用见解的转变，同时减少网络拥塞，简化复杂的分析系统，降低信息技术成本。这个Small Business Technology Transfer (STTR) Phase I项目考察了如何通过允许在存储层中直接处理大型数据对象，然后由客户端应用程序有效地利用这些对象，从而显著加速数据密集型计算问题的挑战。该技术将建立在一个软件定义的数据湖上，用于大数据应用。JSON是使用最广泛的数据交换格式之一，在这个存储解决方案中，将以分布式方式处理JSON，其目标是使客户端分析应用程序能够检索所需内容的子集，而不是完整的对象。存储解决方案将得到增强，以将数据转换为客户端可以直接使用的格式。这将大大提高存储本身以及分析客户端与存储解决方案之间的效率，同时通过绕过瓶颈加速数据处理。其结果将是一个智能数据湖，可以减少网络流量，提高数据新鲜度，实现实时操作，同时将大数据分析速度提高一个数量级或更多。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The broader impact of this Small Business Technology Transfer (STTR) Phase I project will be to fundamentally accelerate the pace of data science. New solutions that alleviate or bypass the bottlenecks inherent in existing data analytics technologies are required to unlock the value contained in “big data” and bring greater benefits to research, business, and society at large. The potential benefits include improved quality control for manufactured goods, reduced fraud in financial transactions, and enhanced customization of consumer services. Many common software applications, such as search engines, online shopping, e-commerce, medical applications, and social networks, are backed by data analytical processing services. This project will enable massive data sets to be pre-processed directly by a shared storage service, then allow this capability to be efficiently utilized by client analytic applications. This project will accelerate transformation of analytical data to useful insights while reducing network congestion, simplifying complex analytics systems, and lowering information technology costs. This Small Business Technology Transfer (STTR) Phase I project examines the challenge of how to dramatically accelerate data-intensive computing problems by enabling large data objects to be processed directly in the storage layer then efficiently utilized by client applications. The technology will be built on a software defined data lake used for big data applications. Key technology will be added to enable JSON, one of the most widely used data interchange formats, to be processed in a distributed manner in-place within this storage solution with the objective of enabling client analytic applications to retrieve not the complete object but only the desired subset of content they require. The storage solution will be augmented to transform the data into a format that can be directly consumed by the clients. This will substantially increase efficiency within the storage itself and between analytics clients and the storage solution – while acceelrating data processing by bypassing bottlenecks. The result will be a smart data lake that can reduce network traffic, improve data freshness, and enable real-time operation – while accelerating big data analytics by an order of magnitude or more.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.