LEEN: Locality/Fairness-Aware Key Partitioning for MapReduce in the Cloud
LEEN: Locality/Fairness-Aware Key Partitioning for MapReduce in the Cloud
复制标题
DOI：
10.1109/cloudcom.2010.25
复制
发表时间：
2010-11
期刊：
影响因子：
--
通讯作者：
Shadi Ibrahim;Hai Jin;Lu Lu-Lu;Song Wu;Bingsheng He;Li Qi
中科院分区：
文献类型：
--
作者：
Shadi Ibrahim;Hai Jin;Lu Lu-Lu;Song Wu;Bingsheng He;Li Qi
本文研究了基于MapReduce的系统中分区偏斜的问题。我们对Hadoop的研究是一种广泛使用的MapReduce实施，表明分区的存在偏斜会在洗牌阶段引起大量数据传输，并导致对不同数据节点之间减少输入的严重不公平。结果，由于在洗牌阶段的长度数据传输以及计算偏斜，尤其是在减少阶段，因此应用程序经历了性能下降。我们开发了一种名为Leen的小说算法，以了解MapReduce中的区域意识和公平意识的关键分区。 Leen包含异步图并减少方案。所有缓冲的中间密钥都根据其频率和休假阶段后预期数据分布的公平性进行分配。我们已将Leen整合到Hadoop-0.18.0中。我们的实验表明，Leen可以有效地实现更高的位置并减少洗牌数据的数量。更重要的是，Leen保证了减少输入的公平分配。结果，Leen在不同的工作量上实现了高达40％的绩效。
This paper investigates the problem of Partitioning Skew in MapReduce-based system. Our studies with Hadoop, a widely used MapReduce implementation, demonstrate that the presence of partitioning skew causes a huge amount of data transfer during the shuffle phase and leads to significant unfairness on the reduce input among different data nodes. As a result, the applications experience performance degradation due to the long data transfer during the shuffle phase along with the computation skew, particularly in reduce phase. We develop a novel algorithm named LEEN for locality-aware and fairness-aware key partitioning in MapReduce. LEEN embraces an asynchronous map and reduce scheme. All buffered intermediate keys are partitioned according to their frequencies and the fairness of the expected data distribution after the shuffle phase. We have integrated LEEN into Hadoop-0.18.0. Our experiments demonstrate that LEEN can efficiently achieve higher locality and reduce the amount of shuffled data. More importantly, LEEN guarantees fair distribution of the reduce inputs. As a result, LEEN achieves a performance improvement of up to 40% on different workloads.