SPX: Collaborative Research: NG4S: A Next-generation Geo-distributed Scalable Stateful Stream Processing System

我们的社会越来越依赖于处理跨地理分布站点的流数据的应用程序，例如根据营销数据做出商业决策，识别社交网络流中的垃圾邮件活动，以及分析不同实验室和国家的基因组数据集以跟踪潜在流行病的来源。满足这些需求的最先进的解决方案是围绕无状态流处理。该项目推进流处理，使下一代流应用程序能够存储和更新状态沿着计算，从而及时处理来自海量和地理分布数据集的实时数据流。现有的系统主要是为数据中心内设置中的无状态流处理而设计的，并且对于运行包含大型分布式状态的流应用程序来说不能很好地扩展。该项目打破了集中式架构和基于哈希表的无状态操作符的传统抽象，用新的分散式架构和新的内存高效的有状态操作符重新定义它们，从而实现了新的方法来提高整体系统性能和可扩展性。 该项目构建了一个下一代地理分布的可扩展的有状态流处理系统，将显着提高流处理系统的可扩展性。这项工作包括三个主要的研究方向。(1)在体系结构层次上，提出了一种新的分散的“多主/多工”体系结构，它为每个主提供了最大的独立性。(2)在操作员级别，将设计和实现一种新的内存数据结构来存储应用程序状态，并最大限度地减少内存开销，以处理“大数据”需求。(3)提出了一种新的基于分片的并行恢复机制，以可扩展的方式处理故障和掉队者。该项目的所有三个部分都将在一个广泛采用的流处理系统（Apache Storm）上原型化和实现。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Our society increasingly relies on applications that process streaming data across geo-distributed sites, such as making business decisions from marketing data, identifying spam campaigns in social network streams, and analyzing genome datasets in different labs and countries to track the sources of potential epidemics. State-of-art solutions for these needs are centered around stateless stream processing. This project advances stream processing to enable next-generation streaming applications to store and update state along with computation, therefore processing live data streams in a timely fashion from massive and geo-distributed datasets. Existing systems are mainly designed for stateless stream processing in intra-datacenter settings and do not scale well for running stream applications that contain large distributed states. This project breaks the traditional abstractions of a centralized architecture and hashtable-based stateless operators, redefining them with a new decentralized architecture and new memory-efficient stateful operators, which enables novel approaches to improve overall system performance and scalability. This project builds a next-generation geo-distributed scalable stateful stream processing system that will significantly improve the scalability of stream processing systems. This work includes three primary research directions. (1) At the architecture level, a new decentralized 'many masters/many workers' architecture will be proposed, which provides each master with maximum independence. (2) At the operator level, a new in-memory data structure will be designed and implemented to store application state and minimize the memory overhead so as to handle 'big data' requirements. (3) A new shard-based parallel recovery mechanism will be proposed to handle failures and stragglers in a scalable way. All three parts of the project will be prototyped and implemented on a widely adopted stream processing system (Apache Storm).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.