Resource optimization for speculative execution in a MapReduce Cluster
Resource optimization for speculative execution in a MapReduce Cluster
复制标题
DOI：
10.1109/icnp.2013.6733646
复制
发表时间：
2013-10
期刊：
影响因子：
--
通讯作者：
Huanle Xu;W. Lau
中科院分区：
文献类型：
--
作者：
Huanle Xu;W. Lau
MapReduce范式现在是大规模数据分析的事实上的标准。本文针对MapReduce机群中的资源管理问题进行了研究。推测执行(任务备份)在资源管理中起着重要的作用。我们提出了两种不同的策略，并建立了两个模型，将集群负载较轻时的备份问题描述为一个优化问题。此外，本文还提出了一种集群负载较高时的增强型推测执行(ESE)算法，并采用近似分析的方法对算法中的参数进行了优化。仿真结果表明，与无备份的朴素算法相比，该算法在减少资源消耗的同时，作业完成时间减少了50%。
The MapReduce paradigm is now the de facto standard for large-scale data analytics. In this paper we address the resource management issues in MapReduce Cluster. Speculative execution (task backup) plays an important role in resource management. We propose two different strategies and build two models to formulate the backup issue as an optimization problem when the cluster is lightly loaded. Moreover, we present an Enhanced Speculative Execution (ESE) algorithm when the cluster is heavily loaded and adopt the approximate analysis to get an optimal value for the parameter in the algorithm. The simulation results show that the algorithm can reduce the job completion time by 50% while consuming much less resource compared to the naive method without backup.