SquirrelJoin: Network-Aware Distributed Join Processing with Lazy Partitioning
SquirrelJoin: Network-Aware Distributed Join Processing with Lazy Partitioning
复制标题
DOI：
10.14778/3137628.3137636
复制
发表时间：
2017-08
期刊：
影响因子：
--
通讯作者：
Lukas Rupprecht;W. Culhane;P. Pietzuch
中科院分区：
文献类型：
--
作者：
Lukas Rupprecht;W. Culhane;P. Pietzuch
要在计算群集上执行分布式加入，与大型数据集之间的系统分区和交换数据记录。带有其他应用程序的带宽描述Squirreljoin，一种分布式的加入处理技术，该技术使用懒惰的分区来适应群体中的瞬态网络偏斜，以维护内存内的懒惰分区，以扣留记录的子集，即不会立即将它们发送给其他工人进行处理。根据网络条件动态分配给其他工人：每个工人进行定期吞吐量测量以估算其完成时间，而懒惰的分区为为了最大程度地减少加入完成时间。 ，固定开销。
To execute distributed joins in parallel on compute clusters, systems partition and exchange data records between workers. With large datasets, workers spend a considerable amount of time transferring data over the network. When compute clusters are shared among multiple applications, workers must compete for network bandwidth with other applications. These variances in the available network bandwidth lead to network skew, which causes straggling workers to prolong the join completion time. We describe SquirrelJoin, a distributed join processing technique that uses lazy partitioning to adapt to transient network skew in clusters. Workers maintain in-memory lazy partitions to withhold a subset of records, i.e. not sending them immediately to other workers for processing. Lazy partitions are then assigned dynamically to other workers based on network conditions: each worker takes periodic throughput measurements to estimate its completion time, and lazy partitions are allocated as to minimise the join completion time. We implement SquirrelJoin as part of the Apache Flink distributed dataflow framework and show that, under transient network contention in a shared compute cluster, SquirrelJoin speeds up join completion times by up to 2.9× with only a small, fixed overhead.