MapReduce Task Scheduling in Heterogeneous Geo-Distributed Data Centers
MapReduce Task Scheduling in Heterogeneous Geo-Distributed Data Centers
复制标题
DOI：
10.1109/tsc.2021.3092563
复制
发表时间：
2022-11
影响因子：
8.1
通讯作者：
Xiaoping Li;Fuchao Chen;Rubén Ruiz;Jie Zhu
中科院分区：
文献类型：
--
作者：
Xiaoping Li;Fuchao Chen;Rubén Ruiz;Jie Zhu
不同的数据传输时间、难以预测的处理时间以及节点相关的访问时间使得MapReduce任务调度变得相当复杂。在本文中，我们考虑了将MapReduce任务调度到分布在不同地理位置的数据中心以最小化总延误的问题。构建了一个新的体系结构来分析所考虑的场景中的数据。我们对不同的数据传输级别、数据中心间和数据中心内以及节点的异构性进行了数学建模。提出了一种在地理位置分散的数据中心中，将MapReduce任务调度到异质节点的算法框架。该算法适用于Hadoop MRv1和MRv2。根据在每个心跳中检测到的空闲容器的数量，从排序的作业序列中选择相同数量的任务。对于映射和约简阶段，分别制定了数据局部性和完成时间的两个度量，并在此基础上采用经典匈牙利算法将选定的任务最优分配到相应的空闲容器。该建议的组成部分和参数是在大量随机实例上进行统计校准的。并将该算法与已有的类似问题的求解方法进行了比较。实验结果表明，该方案对所考虑的问题是有效的。
Different data transmission times, processing times which are difficult to predict and node-dependent access times make MapReduce task scheduling rather complex. In this article, we consider the problem of scheduling MapReduce tasks to heterogeneous geo-distributed data centers to minimize the total tardiness. A new architecture is constructed to analyze data in the considered scenario. We model distinct data transmission levels, inter- and intra- data centers and heterogeneity of nodes mathematically. An algorithm framework is proposed to schedule MapReduce tasks to heterogeneous nodes in geographically distributed data centers. The proposed algorithm is suitable for both Hadoop MRv1 and MRv2. In terms of the number of idle containers detected in each heartbeat, the same number of tasks are selected from a sorted job sequence. For the map and reduce phases, two measurements are developed with data locality and completion time, respectively, based on which the classical Hungarian algorithm is adopted to optimally assign selected tasks to corresponding idle containers. Components and parameters of the proposal are statistically calibrated over a large set of random instances. A comparison of the proposed algorithm to existing methods for similar problems is carried out. Experimental results demonstrate the proposal is effective for the considered problem.