joinTree: A novel join-oriented multivariate operator for spatio-temporal data management in Flink
joinTree: A novel join-oriented multivariate operator for spatio-temporal data management in Flink
复制标题
joinTree：一种新颖的面向连接的多元运算符，用于 Flink 中的时空数据管理
DOI：
10.1007/s10707-022-00470-5
复制
发表时间：
2022-08
期刊：
影响因子：
2
通讯作者：
George Y. Yuan
中科院分区：
文献类型：
--
作者：
Hangxu Ji;Gang Wu;Yuhai Zhao;Shiye Wang;Guoren Wang;George Y. Yuan
智能互联网时代，海量时空数据的管理与分析是实现智能化应用、建设智慧城市的重要环节之一，其中多源数据的交互是实现时空数据管理与分析的基础。 Flink 作为实现海量数据交互计算的重要载体，提供了先进的 Operator Join 来方便用户程序开发。在多源数据连接操作的Flink作业中，Join序列的选择以及重分区阶段的数据通信都是影响作业效率的关键因素。然而，Flink 并没有针对这两个因素提供任何优化机制，从而导致工作效率较低。如果采用枚举法寻找最优连接序列，则无法在多项式时间内得到结果，因此达不到优化效果。我们研究了上述问题，在 Flink 中设计并实现了一个更先进的可以支持多源数据连接的 Operator joinTree，并在 Operator 中引入了两种优化策略。综上，我们工作的优势突出如下：（1）Operator使得Flink支持多源数据连接操作，通过引入轻量级优化策略减少计算量和数据通信量，提高工作效率； (2)采用Join顺序优化策略，与传统顺序执行相比，总运行时间可减少29%，数据通信可减少34%； (3)数据重新分区的优化策略可以进一步使作业带来35%的性能提升，并且在平均情况下可以减少43%的数据通信。
In the era of intelligent Internet, the management and analysis of massive spatio-temporal data is one of the important links to realize intelligent applications and build smart cities, in which the interaction of multi-source data is the basis of realizing spatio-temporal data management and analysis. As an important carrier to achieve the interactive calculation of massive data, Flink provides the advanced Operator Join to facilitate user program development. In a Flink job with multi-source data connection operations, the selection of join sequences and the data communication in the repartition phase are both key factors that affect the efficiency of the job. However, Flink does not provide any optimization mechanism for the two factors, which in turn leads to low job efficiency. If the enumeration method is used to find the optimal join sequence, the result will not be obtained in polynomial time, so the optimization effect cannot be achieved. We investigate the above problems, design and implement a more advanced Operator joinTree that can support multi-source data connection in Flink, and introduce two optimization strategies into the Operator. In summary, the advantages of our work are highlighted as follows: (1) the Operator enables Flink to support multi-source data connection operation, and reduces the amount of calculation and data communication by introducing lightweight optimization strategies to improve job efficiency; (2) with the optimization strategy for join sequence, the total running time can be reduced by 29% and the data communication can be reduced by 34% compared with traditional sequential execution; (3) the optimization strategy for data repartition can further enable the job to bring 35% performance improvement, and in the average case can reduce the data communication by 43%.
登录
查看更多内容
DOI：
10.1109/icde.2019.00086
发表时间：
2019-04
期刊：
2019 IEEE 35th International Conference on Data Engineering (ICDE)
影响因子：
--
作者：
Ye Yuan;Xiang Lian;Guoren Wang;Lei Chen;Yuliang Ma;Yishu Wang
通讯作者：
Ye Yuan;Xiang Lian;Guoren Wang;Lei Chen;Yuliang Ma;Yishu Wang
DOI：
--
发表时间：
2010-06
期刊：
--
影响因子：
--
作者：
M. Zaharia;Mosharaf Chowdhury;Michael J. Franklin;S. Shenker;I. Stoica
通讯作者：
M. Zaharia;Mosharaf Chowdhury;Michael J. Franklin;S. Shenker;I. Stoica
影响因子：
8.9
作者：
F. Afrati;J. Ullman
通讯作者：
F. Afrati;J. Ullman
DOI：
10.1145/1989323.1989328
发表时间：
2011-06
期刊：
--
影响因子：
--
作者：
Spyros Blanas;Yinan Li;J. Patel
通讯作者：
Spyros Blanas;Yinan Li;J. Patel
DOI：
--
发表时间：
1996
期刊：
--
影响因子：
--
作者：
Thomas Sttzle;H. Hoos
通讯作者：
Thomas Sttzle;H. Hoos