MRTuner: A Toolkit to Enable Holistic Optimization for MapReduce Jobs
MRTuner: A Toolkit to Enable Holistic Optimization for MapReduce Jobs
复制标题
DOI：
10.14778/2733004.2733005
复制
发表时间：
2014-08
期刊：
影响因子：
--
通讯作者：
Juwei Shi;Jia Zou;Jiaheng Lu;Zhao Cao;Shiqiang Li;Chen Wang
中科院分区：
文献类型：
--
作者：
Juwei Shi;Jia Zou;Jiaheng Lu;Zhao Cao;Shiqiang Li;Chen Wang
基于MapReduce的数据密集型计算解决方案越来越多地部署为生产系统。与互联网公司从一开始就发明和采用该技术不同，传统企业由于管理员的能力有限，需要易于使用的软件。MapReduce的自动作业优化软件是一种很有前途的技术，以满足这些要求。在本文中，我们介绍了IBM的一个工具包，称为MRTuner，使MapReduce作业的整体优化。特别是，我们提出了一种新的生产者-运输者-消费者（PTC）模型，其特征在于在任务之间的并行执行的权衡。我们还仔细研究了约二十个参数之间的复杂关系，这些参数对工作绩效有重要影响。我们设计了一个有效的搜索算法来找到最优的执行计划。最后，我们使用HiBench套件对两种不同类型的集群进行了全面的实验评估，该套件涵盖了从GB到TB大小级别的各种Hadoop工作负载。结果表明，MRTuner的搜索延迟比目前最先进的基于代价的优化器快了几个数量级，优化后的执行计划的有效性也有了显著提高。
MapReduce based data-intensive computing solutions are increasingly deployed as production systems. Unlike Internet companies who invent and adopt the technology from the very beginning, traditional enterprises demand easy-to-use software due to the limited capabilities of administrators. Automatic job optimization software for MapReduce is a promising technique to satisfy such requirements. In this paper, we introduce a toolkit from IBM, called MRTuner, to enable holistic optimization for MapReduce jobs. In particular, we propose a novel Producer-Transporter-Consumer (PTC) model, which characterizes the tradeoffs in the parallel execution among tasks. We also carefully investigate the complicated relations among about twenty parameters, which have significant impact on the job performance. We design an efficient search algorithm to find the optimal execution plan. Finally, we conduct a thorough experimental evaluation on two different types of clusters using the HiBench suite which covers various Hadoop workloads from GB to TB size levels. The results show that the search latency of MRTuner is a few orders of magnitude faster than that of the state-of-the-art cost-based optimizer, and the effectiveness of the optimized execution plan is also significantly improved.