Collaborative Research:PPoSS:Planning: Streamware - A Scalable Framework for Accelerating Streaming Data Science

在具有重大挑战的科学应用中，传感和仪器基础设施产生的大量数据往往在一小段时间后失去其价值。因此，为了从数据中获得可操作的情报，流分析（即分析动态数据的能力）变得越来越重要。此外，现代计算系统是高度异构的，由处理器、加速器和大型高带宽外部存储器组成。为了开发可扩展的流分析应用程序，需要解决从应用程序到目标平台的整个系统堆栈的挑战。在这方面，该规划项目正在确定一套全面的研究挑战、目标、关键创新和时间表，涉及算法和应用、系统软件、硬件软件协同设计和计算机体系结构。该项目汇集了一个由应用程序开发人员和用户、计算机科学家和数据科学家组成的社区，他们的兴趣在于构建针对各种可扩展系统的流数据科学应用程序。该项目正在展示如何使用用于安全智能电网的隐私保护流图学习作为驱动应用程序来实现显著的跨堆栈性能改进的初步结果。现代数据科学应用的特点是高度分散、分布式，需要在数千或数百万个边缘平台的本地化分析和云/数据中心的大规模集中分析之间进行组合和编排，以及需要对流数据进行实时分析。为了实现具有重大挑战的流数据科学应用程序的可伸缩性能，需要一个框架，允许开发人员无缝地构建针对各种可伸缩系统的这些应用程序。这个计划项目正在为开发一个大型开源框架StreamWare进行初步研究，该框架将使用户能够开发流数据科学应用程序。该项目正在建立一个由应用程序开发人员和用户、计算机科学家和数据科学家组成的社区，他们将成为StreamWare框架的早期采用者和开发人员。在咨询了领域专家后，我们为StreamWare生成了一份关键数据科学内核清单，并对其现有的最先进算法和硬件ip进行了评估，以确定性能限制和改进机会。该项目还阐明了能够在异构平台上表示和操作流数据的新颖抽象的需求。该项目使用隐私保护流图学习作为安全智能电网的激励应用程序，使用新的共生可扩展性概念来展示端到端可扩展性的初步证据，该概念捕获了StreamWare跨层优化的影响。此规划项目的预期成果包括一份将在大笔拨款内进行的研究活动的建议、有关调查活动结果的出版物和未来流数据科学的研究方向，以及未来研究生和本科课程的课程。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

In grand-challenge scientific applications, the enormous amount of data produced by the sensing and instrumentation infrastructure often loses its value after a small window of time. Thus, to obtain actionable intelligence from the data, streaming analytics, i.e., the ability to analyze in-motion data, is increasingly becoming critical. Moreover, modern computing systems are highly heterogeneous, consisting of processors, accelerators, and large high-bandwidth external memories. To develop scalable streaming analytics applications, challenges across the full system stack -- from application to target platform -- need to be addressed. In this regard, this planning project is identifying a comprehensive set of research challenges, goals, key innovations and timelines in algorithms and applications, systems software, hardware-software co-design, and computer architecture. This project is bringing together a community of application developers and users, computer scientists, and data scientists, whose interests lie in building streaming data science applications targeting a wide variety of scalable systems. This project is demonstrating preliminary results on how it will achieve significant cross-stack performance improvements using Privacy Preserving Streaming Graph Learning for Secure Smart Grids as the driving application.Modern data-science applications are characterized as being highly decentralized, distributed and requiring composition and orchestration between localized analytics on thousands or millions of edge platforms and massive centralized analytics in cloud/data centers, as well as requiring real-time analytics on streaming data. To enable scalable performance of grand-challenge streaming data-science applications, a framework that allows developers to seamlessly build these applications targeting a wide variety of scalable systems is needed. This planning project is conducting preliminary research towards a large proposal for developing an opensource framework, StreamWare, that will enable users to develop streaming data-science applications. This project is establishing a community of application developers and users, computer scientists, and data scientists who would serve as early adopters and developers of the StreamWare framework. In consultation with domain experts, a list of key data-science kernels for StreamWare is being generated, and their existing state-of-the-art algorithms and hardware IPs are being evaluated to identify performance limitations and opportunities for improvement. This project is also articulating the requirements of novel abstractions that can represent and operate on streaming data on heterogeneous platforms. This project uses Privacy Preserving Streaming Graph Learning for Secure Smart Grids as a motivating application to show preliminary evidence of end-to-end scalability using a novel notion of symbiotic scalability that captures the impact of StreamWare's cross-layer optimizations. The expected outcomes of this planning project include a proposal for the research activities to be carried out in the large grant, publications on the results of the survey activities and future research directions for enabling streaming data science, and curricula for future graduate and undergraduate courses.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.