Automating Characterization Deployment in Distributed Data Stream Management Systems
Automating Characterization Deployment in Distributed Data Stream Management Systems
复制标题
分布式数据流管理系统中的自动化表征部署
DOI：
10.1109/tkde.2017.2751606
复制
发表时间：
2017
影响因子：
8.9
通讯作者：
Yang Chen
中科院分区：
文献类型：
--
作者：
Wang Chunkai;Meng Xiaofeng;Guo Qi;Weng Zujian;Yang Chen
分布式数据流管理系统（DDSMS）通常由上层关系查询系统（RQS）和下层流处理系统（SPS）组成。当用户向RQS提交新的查询时，需要将查询规划器转换为由SPS上运行的任务组成的有向无环图（DAG）。根据不同的查询请求和数据流属性，SPS需要配置不同的部署策略。然而，如何动态预测SPS的部署配置，以确保处理吞吐量和低资源使用率是一个巨大的挑战。本文介绍了OrientStream，这是一个使用增量机器学习技术在DDSMS中自动化特征部署的框架。通过引入数据级、查询计划级、操作员级和簇级的4级特征提取机制，首先利用不同的查询负载作为训练集，预测DDSMS的资源使用情况，并根据当前的查询请求和流属性从候选设置中选择最优的资源配置，然后引入动态重配置迁移操作员状态.最后，我们在开源SPS-Storm上验证了我们的方法。针对监控周期长、数据波动不频繁的应用场景，实验表明，OrientStream可以分别降低8- 15%的CPU使用率和38- 48%的内存使用率。
Distributed data stream management systems (DDSMS) are usually composed of upper layer relational query systems (RQS) and lower layer stream processing systems (SPS). When users submit new queries to RQS, a query planner needs to be converted into a directed acyclic graph (DAG) consisting of tasks which are running on SPS. Based on different query requests and data stream properties, SPS need to configure different deployments strategies. However, how to dynamically predict deployment configurations of SPS to ensure the processing throughput and low resource usage is a great challenge. This article presents OrientStream, a framework for automating characterization deployment in DDSMS using incremental machine learning techniques. By introducing the data-level, query plan-level, operator-level, and cluster-level’s four-level feature extraction mechanism, we first use the different query workloads as training sets to predict the resource usage by DDSMS, and select the optimal resource configuration from candidate settings based on the current query requests and stream properties, then migrate the operator state by introducing dynamic reconfiguration. Finally, we validate our approach on the open source SPS–Storm. In view of the application scenarios with long monitoring cycle and non-frequent data fluctuation, experiments show that OrientStream can reduce CPU usage of 8-15 percent and memory usage of 38-48 percent, respectively.