SR3: Customizable Recovery for Stateful Stream Processing Systems
SR3: Customizable Recovery for Stateful Stream Processing Systems
复制标题
DOI：
10.1145/3423211.3425681
复制
发表时间：
2020-12
期刊：
影响因子：
--
通讯作者：
Hailu Xu;Pinchao Liu;Susana Cruz-Diaz;D. D. Silva-D.;Liting Hu
中科院分区：
文献类型：
--
作者：
Hailu Xu;Pinchao Liu;Susana Cruz-Diaz;D. D. Silva-D.;Liting Hu
现代流处理应用程序需要存储和更新状态及其处理，并及时从大型和地理分布的数据集中处理实时数据流。由于它们在动态的分布环境中运行，而且工作负载可能会以意外的方式发生变化，因此多个流操作员可能同时失败，从而导致严重的州损失。但是，最先进的流处理系统主要是为低延迟的纳入式设置设计的，对于包含大型分布式状态的运行流应用程序，遭受了相当大的集中式瓶颈和高延迟恢复状态，并不能很好地扩展。 。他们主要通过三种方法提供故障恢复：复制恢复，检查点恢复和基于Dstream的谱系恢复，它们要么缓慢，昂贵，要么无法处理多个同时失败。我们提出SR3，这是一个可自定义的状态恢复框架，可提供快速，可扩展的状态恢复机制，用于保护流处理系统中的大型分布式状态。 SR3提供了三种恢复机制---恒星结构恢复，线结构恢复和树结构化恢复 - 以满足不同流处理计算模型，状态大小和网络设置的需求。我们的设计采用了分散的体系结构，通过使用一致的环形覆盖层来分配和复制状态，以利用分布式哈希表（DHTS）。我们表明，这种方法可以显着提高状态恢复的可扩展性和灵活性。我们在与广泛采用的Apache风暴框架集成的原型上实现了SR3设计。使用现实世界数据集的大规模实验证明了SR3的可扩展性，快速恢复和灵活性属性。
Modern stream processing applications need to store and update state along with their processing, and process live data streams in a timely fashion from massive and geo-distributed data sets. Since they run in a dynamic distributed environment and their workloads may change in unexpected ways, multiple stream operators can fail at the same time, causing severe state loss. However, the state-of-the-art stream processing systems are mainly designed for low-latency intra-datacenter settings and do not scale well for running stream applications that contain large distributed states, suffering a significantly centralized bottleneck and high latency to recover state. They offer failure recovery mainly through three approaches: replication recovery, checkpointing recovery, and DStream-based lineage recovery, which are either slow, resource-expensive or fail to handle multiple simultaneous failures. We present SR3, a customizable state recovery framework that provides fast and scalable state recovery mechanisms for protecting large distributed states in stream processing systems. SR3 offers three recovery mechanisms --- the star-structured recovery, the line-structured recovery, and the tree-structured recovery --- to cater to the needs of different stream processing computation models, state sizes, and network settings. Our design adopts a decentralized architecture that partitions and replicates states by using consistent ring overlays that leverage distributed hash tables (DHTs). We show that this approach can significantly improve the scalability and flexibility of state recovery. We realize the SR3 design on a prototype integrated with the widely adopted Apache Storm framework. Large-scale experiments using real-world datasets demonstrate SR3's scalability, fast recovery, and flexibility properties.