Throughput optimization for Storm-based processing of stream data on clouds
Throughput optimization for Storm-based processing of stream data on clouds
复制标题
基于Storm的云上流数据处理吞吐量优化
DOI：
10.1016/j.future.2020.06.009
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
Shen, Wei
中科院分区：
文献类型：
--
作者：
Cao, Huiyan;Wu, Chase Q.;Bao, Liang;Hou, Aiqin;Shen, Wei
在各种大数据应用中，对实时处理大量流数据的需求正在迅速增长。作为最常用的流数据处理系统之一，Apache Storm 提供了一种基于工作流的机制来执行有向无环图 (DAG) 结构的拓扑。随着全球云基础设施的扩展以及基于云的计算和存储服务的经济效益，许多此类 Storm 工作流程已经转移或正在积极过渡到云。然而，对流数据处理的行为进行建模并提高其在云中的性能仍然很大程度上尚未得到探索。我们构建严格的成本模型来分析 Storm 工作流的吞吐量动态，并制定预算受限的拓扑映射问题，以最大限度地提高云中 Storm 工作流的吞吐量。我们证明这个问题是 NP 完全的，并设计了一个启发式解决方案，该解决方案不仅考虑了虚拟机类型的选择，还考虑了拓扑中每个任务（spout/bolt）的并行度。与默认的 Storm 和其他现有方法相比，所提出的映射解决方案的性能优越性通过广泛的模拟得到了证明，并通过部署在公共云中的实际工作流实验得到了进一步验证。
There is a rapidly growing need for processing large volumes of streaming data in real time in various big data applications. As one of the most commonly used systems for streaming data processing, Apache Storm provides a workflow-based mechanism to execute directed acyclic graph (DAG)-structured topologies. With the expansion of cloud infrastructures around the globe and the economic benefits of cloud-based computing and storage services, many such Storm workflows have been shifted or are in active transition to clouds. However, modeling the behavior of streaming data processing and improving its performance in clouds still remain largely unexplored. We construct rigorous cost models to analyze the throughput dynamics of Storm workflows and formulate a budget-constrained topology mapping problem to maximize Storm workflow throughput in clouds. We show this problem to be NP-complete and design a heuristic solution that takes into consideration not only the selection of virtual machine type but also the degree of parallelism for each task (spout/bolt) in the topology. The performance superiority of the proposed mapping solution is illustrated through extensive simulations and further verified by real-life workflow experiments deployed in public clouds in comparison with the default Storm and other existing methods.
优化异构网络环境中数据密集型计算管道的端到端性能
DOI：
10.1016/j.jpdc.2010.08.003
发表时间：
2011
期刊：
J. Parallel Distributed Comput.
影响因子：
--
作者：
C. Wu;Yi Gu
通讯作者：
Yi Gu