WASP: Wide-area Adaptive Stream Processing
WASP: Wide-area Adaptive Stream Processing
复制标题
DOI：
10.1145/3423211.3425668
复制
发表时间：
2020-12
期刊：
影响因子：
--
通讯作者：
A. Jonathan;A. Chandra;J. Weissman
中科院分区：
文献类型：
--
作者：
A. Jonathan;A. Chandra;J. Weissman
对于流处理系统来说，适应性是确保稳定、低延迟和高吞吐量处理长时间运行的查询的关键。由于广域环境的高度动态性质，这种适应性对于广域流处理是特别具有挑战性的，广域环境包括不可预测的工作负载模式、可变的网络带宽、掉队者的发生和故障。不幸的是，现有的自适应技术通常通过损害结果的质量/准确性来实现这些性能目标，并且它们通常是依赖于应用的。在这项工作中，我们重新思考广域流处理系统的适应性，并提出了一个资源感知的自适应框架，称为WASP。WASP通过多种技术的组合来适应查询：任务重新分配，操作员缩放和查询重新规划，并以WAN感知的方式应用它们。它能够根据查询类型、动态和优化目标自动确定要采取的自适应操作。我们已经在Apache Flink上实现了一个WASP原型。使用YSB基准测试和真实的Twitter跟踪进行的实验评估表明，WASP可以处理各种动态，而不会影响结果的质量。
Adaptability is critical for stream processing systems to ensure stable, low-latency, and high-throughput processing of long-running queries. Such adaptability is particularly challenging for wide-area stream processing due to the highly dynamic nature of the wide-area environment, which includes unpredictable workload patterns, variable network bandwidth, occurrence of stragglers, and failures. Unfortunately, existing adaptation techniques typically achieve these performance goals by compromising the quality/accuracy of the results, and they are often application-dependent. In this work, we rethink the adaptability property of wide-area stream processing systems and propose a resource-aware adaptation framework, called WASP. WASP adapts queries through a combination of multiple techniques: task re-assignment, operator scaling, and query re-planning, and applies them in a WAN-aware manner. It is able to automatically determine which adaptation action to take depending on the type of queries, dynamics, and optimization goals. We have implemented a WASP prototype on Apache Flink. Experimental evaluation with the YSB benchmark and a real Twitter trace shows that WASP can handle various dynamics without compromising the quality of the results.