STTR Phase I: Real-Time Smart Data Lake

小企业技术转让（STTR）第一阶段项目的更广泛影响将从根本上加快数据科学的步伐。需要缓解或绕过现有数据分析技术固有瓶颈的新解决方案，以释放“大数据”中包含的价值，并为研究，商业和整个社会带来更大的利益。 潜在的好处包括改进制成品的质量控制，减少金融交易中的欺诈行为，以及加强消费者服务的定制化。 许多常见的软件应用程序，如搜索引擎、在线购物、电子商务、医疗应用程序和社交网络，都由数据分析处理服务提供支持。该项目将使大量数据集能够直接由共享存储服务进行预处理，然后允许客户端分析应用程序有效地利用此功能。该项目将加速分析数据转化为有用的见解，同时减少网络拥塞，简化复杂的分析系统，并降低信息技术成本。此小型企业技术转让（STTR）第一阶段项目探讨了如何通过使大型数据对象能够直接在存储层中处理，然后由客户端应用程序有效利用，从而大幅加速数据密集型计算问题的挑战。该技术将建立在用于大数据应用程序的软件定义数据湖上。将添加关键技术，以使JSON（最广泛使用的数据交换格式之一）能够在此存储解决方案中以分布式方式进行处理，目的是使客户端分析应用程序能够检索所需的内容子集，而不是完整的对象。存储解决方案将得到增强，以将数据转换为客户端可以直接使用的格式。这将大大提高存储本身以及分析客户端和存储解决方案之间的效率，同时通过绕过瓶颈来加速数据处理。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响力审查标准进行评估，被认为值得支持。

The broader impact of this Small Business Technology Transfer (STTR) Phase I project will be to fundamentally accelerate the pace of data science. New solutions that alleviate or bypass the bottlenecks inherent in existing data analytics technologies are required to unlock the value contained in “big data” and bring greater benefits to research, business, and society at large. The potential benefits include improved quality control for manufactured goods, reduced fraud in financial transactions, and enhanced customization of consumer services. Many common software applications, such as search engines, online shopping, e-commerce, medical applications, and social networks, are backed by data analytical processing services. This project will enable massive data sets to be pre-processed directly by a shared storage service, then allow this capability to be efficiently utilized by client analytic applications. This project will accelerate transformation of analytical data to useful insights while reducing network congestion, simplifying complex analytics systems, and lowering information technology costs. This Small Business Technology Transfer (STTR) Phase I project examines the challenge of how to dramatically accelerate data-intensive computing problems by enabling large data objects to be processed directly in the storage layer then efficiently utilized by client applications. The technology will be built on a software defined data lake used for big data applications. Key technology will be added to enable JSON, one of the most widely used data interchange formats, to be processed in a distributed manner in-place within this storage solution with the objective of enabling client analytic applications to retrieve not the complete object but only the desired subset of content they require. The storage solution will be augmented to transform the data into a format that can be directly consumed by the clients. This will substantially increase efficiency within the storage itself and between analytics clients and the storage solution – while acceelrating data processing by bypassing bottlenecks. The result will be a smart data lake that can reduce network traffic, improve data freshness, and enable real-time operation – while accelerating big data analytics by an order of magnitude or more.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.