Handling Data Skew in MapReduce Cluster by Using Partition Tuning.
Handling Data Skew in MapReduce Cluster by Using Partition Tuning.
复制标题
使用分区调优处理MapReduce集群中的数据倾斜
DOI：
10.1155/2017/1425102
复制
发表时间：
2017
影响因子：
--
通讯作者：
Zhang J
中科院分区：
文献类型：
--
作者：
Gao Y;Zhou Y;Zhou B;Shi L;Zhang J
医疗保健行业产生了大量数据，分析这些数据已成为近年来的一个重要问题。MapReduce编程模型已成功用于大数据分析。然而，在大数据分析中，数据倾斜不可避免地会发生，严重影响效率。为了克服MapReduce中的数据倾斜问题，我们在过去提出了一种称为基于分区调优的倾斜处理(PTSH)的数据处理算法。与传统MapReduce模型中使用的一阶段分区策略相比，PTSH使用两阶段策略和分区调优方法将键值对分散在虚拟分区中，并在数据倾斜的情况下对每个分区进行重组。在各种模拟数据集和真实医疗数据集上测试了该算法的稳健性和效率。实验结果表明，与原有的Hadoop、Closer、局部感知和公平感知的密钥划分(LEEN)算法相比，PTSH算法能够有效地处理MapReduce中的数据倾斜问题，并提高了MapReduce作业的性能。我们还发现，采用PTSH算法可以显著减少规则提取所需的时间，因为它更适合于医疗数据的关联规则挖掘(ARM)。
The healthcare industry has generated large amounts of data, and analyzing these has emerged as an important problem in recent years. The MapReduce programming model has been successfully used for big data analytics. However, data skew invariably occurs in big data analytics and seriously affects efficiency. To overcome the data skew problem in MapReduce, we have in the past proposed a data processing algorithm called Partition Tuning-based Skew Handling (PTSH). In comparison with the one-stage partitioning strategy used in the traditional MapReduce model, PTSH uses a two-stage strategy and the partition tuning method to disperse key-value pairs in virtual partitions and recombines each partition in case of data skew. The robustness and efficiency of the proposed algorithm were tested on a wide variety of simulated datasets and real healthcare datasets. The results showed that PTSH algorithm can handle data skew in MapReduce efficiently and improve the performance of MapReduce jobs in comparison with the native Hadoop, Closer, and locality-aware and fairness-aware key partitioning (LEEN). We also found that the time needed for rule extraction can be reduced significantly by adopting the PTSH algorithm, since it is more suitable for association rule mining (ARM) on healthcare data.